Kitty Corp Meow Mix Forums

Super Smash Bros. Brawl Hacking => Programming => Topic started by: Xiggah on January 23, 2012, 11:26:26 PM



Title: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 23, 2012, 11:26:26 PM
In this thread, we will train ourselves to parse Brawl's files, so Brawlbox will be able to edit more files and be capable of more hacks. In layman's terms, parsing a file means sorting out different parts of the file, similar to separating stacked lego blocks. We can do this through hexing Brawl's files and finding patterns in its code. There will be more tutorials, but until then feel free to use this thread to ask questions about file parsing and hexing.

Here's a short list of file types that need parsed:
Files needing parsed:
- BLOC     File container, like ARC or BRRES, used commonly in Subspace Emissary files.
- REFF     Normally seen in files with AI data, such as CPU controlled characters and items.
- SCL0     No idea?...
- SHP0     (SHaPe) Vertex morph, an animation file used to change a character's facial expression.
- PAT0     (PATtern) Texture pattern, an animation file that switches textures to simulate an animated texture.
- Havok Physics     Physics engine files, used for many things including hair and cape physics.

Tutorials

    Intro to Hexing
For those completely new to hexing, let's do a quick little exercise. Open Brawlbox, select File --> New --> BRRES. Right-Click the BRRES and select New --> Character Animation. Finally, Right-Click the animation and create a new bone animation. You should end up with this:
(http://i.imgur.com/BqVqv.png)

Now you should have a basic animation file, 1 frame without any animation data. Right-Click and export the NEWCHR animation (CHR0) so it can be hexed. Open up the animation CHR0 in your hex editor, any hex editor will work, I'll be using Notepad++ but the results will be the same. Assuming you didn't rename the CHR0, it should look more or less like this, depending on your view settings:
(http://i.imgur.com/bU5mX.png)

So you might be thinking "i could never read what these random numbers mean!" but let's get into that right now. Since there aren't any animations in this file, it's pretty much empty. Open the animation in Brawlbox and change the Translation X keyframe value to 1, export the CHR0 and open it in hex again.
(http://i.imgur.com/wu4FQ.png)

You can see some changes here and there, but how do we find out what they mean? Thanks to Brawlbox, we know that this animation file has a single keyframe with a value of 1. That number (keyframe value, not amount of keyframes) was translated into a 8-digit value called a Float value, which can be translated with this webpage (http://www.h-schmidt.net/FloatApplet/IEEE754.html). Tools like these are a hexer's best friend.

Using that webpage, you can see that 1.0 in hexidecimal is 3F 80 00 00. You should be able to find that number in your hex editor: if not, then It's highlighted in the previous image. Any single number, even 99999 or 3.14, will be displayed as an 8-digit float value. If you change the keyframe value in Brawlbox, then the only difference will be that float value: everything else remains the same.

Now let's do something cool with your hex editor: Click on the beginning of the float value and type 44 a7 20 00, save and open in Brawlbox. You just hexed the keyframe value from 1 to 1337. Pretty neat, right?

This is what Brawl hackers do to change simple number values that aren't editable in Brawlbox, as well as hacking computer files in general. Try using your new knowledge of float values and hexing to change numbers in other Brawl files, or add more keyframes with numbers and see where each number ends up.

In the meantime, I'll set up the next tutorial about identifying file structures.

Download links:
HXD (http://mh-nexus.de/en/downloads.php?product=HxD), a commonly-used hex editor.
Notepad++ (http://notepad-plus-plus.org/download/v5.9.8.html), a handy program based off of Notepad with a Hex Plugin (http://sourceforge.net/projects/npp-plugins/files/Hex%20Editor/Hex%20Editor%20Plugin%20v0.9.4/HexEditor_0_9_4_UNI_dll.zip/download?use_mirror=superb-dca2).
Hexidecimal Number Converter (http://www.h-schmidt.net/FloatApplet/IEEE754.html) (website, not a download)


File structures:

BLOC
0x00(4) - BLOC
0x04(4) - # of entries
0x08(4) - Unknown
0x0C(4) - Nulls?

Offsets (starts at 0x10)
4 bytes * # of entries = size of string table

DAT (Event Data)
Header Size - 0x50

0x08 = Match Type(Value is at 2 for Coin event match)
0x0C = Time Limit
0x1F = Stage ID
0x38 = Game Speed
0x3C = Camera Shaking control
0x46 = Music ID

PlayerData Size = 0x38
---------------------------------
0x00  = byte CharacterID
0x01  = byte Status (Normal,Metal,Invisible,BEEP?)
0x04  = float Size(Scaling)
0x08  = Team Flag
0x0C  = ?????(Easy)
0x0F  = Offense Ratio(Easy)
0x11  = Defense Ratio(Easy)
0x14  = Stock count(Easy)
0x18  = Starting Damage(Easy)
0x1B  = ?????(Normal)
0x1D  = Offense Ratio(Normal)
0x1F  = Defense Ratio(Normal)
0x22  = Stock count(Normal)
0x26  = Starting Damage(Normal)
0x28  = ?????(Hard)
0x2B  = Offense Ratio(Hard)
0x2D  = Defense Ratio(Hard)
0x30  = Stock count(Hard)
0x34  = Starting Damage(Hard)

SHP0
unsafe struct SHP0
    {
        public BRESCommonHeader _header;
        public bint _dataOffset;
        public bint _stringListOffset; //List of vertex node strings
        public bint _stringOffset;
        public bint _unk1; //0
        public bshort _numFrames;
        public bshort _numEntries;
        public bint _unk2; //0x00, 0x01

unsafe struct SHP0Entry
    {
        public bint _numEntries; //entries + 1 index part
        public bint _stringOffset;
        public bint _numIndices;
        public bint _unk;
        public bint _indiciesOffset;

I'm pretty sure the keyframes are I12 Keyframe Entries except with no frame scale.

    public unsafe struct I12Header
    {
        public bushort _numFrames;
        public bushort _unk;
        public bfloat _frameScale; //This is the value that isn't present in SHP0

public unsafe struct I12Entry
    {
        public bfloat _index;
        public bfloat _value;
        public bfloat _tangent;

Havok Physics
public unsafe struct PhysicsHeader
    {
        public const uint Tag1 = 0x57E0E057;
        public const uint Tag2 = 0x10C0C010;

        public uint _tag1; //0x57E0E057
        public uint _tag2; //0x10C0C010
        public bint _unk2; //0
        public bint _unk3; //4

        public byte _unk4; //4
        public bshort _unk5; //1
        public byte _unk6; //1
        public bint _unk7; //3
        public bint _unk8; //1
        public bint _unk9; //0

        public bint _unk10; //0
        public buint _unk11; //Size/Offset

        //Three sections of offsets:
        //__classnames__
        //__data__
        //__types__

        public OffsetSection ClassNames;
        public OffsetSection Data;
        public OffsetSection Types;

        public String Name { get { return new String((sbyte*)Address + 0x28); } }
        
        public VoidPtr ClassNamesData { get { return Address + *(buint*)(Address + 0x54); } }
        public VoidPtr DataData { get { return Address + *(buint*)(Address + 0x84); } }
        public VoidPtr TypesData { get { return Address + *(buint*)(Address + 0xB4); } }
}

public unsafe struct OffsetSection
    {
        public fixed byte name[0x10];

        public bint _unk0; //-1
        public buint _dataOffset; //Main header is the base
        
        //Offsets to indices struct. _dataOffset is the base for everything.
        public buint _dataOffset1;
        public buint _dataOffset2;
        public buint _dataOffset3;
        public buint _dataOffset4;
        public buint _dataOffset5;
        public buint _dataOffset6;

        //When offsets begin to repeat, stop reading
        //Indices count is ((next data offset) - (current data offset)) / 2
        //Indices are padded to 0x10 with 0xFF

        public String Name { get { return new String((sbyte*)Address); } }
    }

public unsafe struct IndexGroup
    {
        //Base is the offset sections's _dataOffset
        //Still not sure exactly how these are used, but they're definitely a datalength or offset
        
        public buint _dataOffset; //Not necessarily...
        public buint _stringOffset; //Not necessarily...

PAT0
Header:
0x00 (4) PAT0
0x04 (4) String Offset Table
0x08 (4) Version
0x0C (4) Unknown1
0x10 (4) Pipeline offset
0x14 (4) "Texture Reference" (will describe this later)
0x18 (4) Unknown2, one offset that leading to a emtpy space
0x1C (4) Unknown3, often the same as Unknown2
0x20 (4) String Offset Table (Copy)
0x24 (4) Name of animation Offset
0x28 (4) Unknown4
0x2C (2) Frames
0x2E (2) NumEntries/Materials
0x30 (2) How many Textures (String Entries1)
0x32 (2) String Entries 2, often set as "00"
0x34 (4) Unknown5

PipeLine:
0x00 (4) How long PipeLine is
0x04 (4) How many Entries/IDs
0x08-1C IDs thingy
dont exactly know how the ID's working but this is importiant:
0x20 (4) It's name in String Offset (Pipelines offset + this offset = string offset)
0x24 (4) "link" offset (Pipeline offset + this offset = "Link" offset)
and the ID and Stirng/pattern offsets repeat on each entries

"Link" Offset:
0x00 (4) String Offset, linked to Materials name
0x04 (2) What Texture it will using if Flag is set on "7". if the flag is "5" this is set on "00"
0x06 (2) Flag: 5 using Pattern, 7 is Frozen
0x08 (4) Pattern Animation Offset (Link Offset + this = Pattern Offset). this is set on "00" if the Flag is set on "7"

Pattern Animation offset
0x00 (2) Unknown, possibly flag
0x02 (2) Unknown
0x04 (4)  unknown, Possible many flags in "bits"
0x08 (4) unknown, always(?) set on 00
0x0C (2) Which Texture will be shown
0x0E (2) Unknown, always(?) set on 00
0x10 (2) Which frame and possible with a flag with it (if it saying 41B8, the 4 is a flag and 1B8 is frame)

Repeat from 0x0C to 0x10 on how many frames

"Texture refserence" Offset
here is special and hard to explain, the textures here comes in order thanks by the "link" offset and it's the order the Pattern Animation in their "texture"

0x00 (4) Offset to their texture name in String Offset

String Table Offset:
well, all the names are in here, always in the end of the PAT0s


Title: Re: File Parsing Boot Camp
Post by: Xiggah on January 23, 2012, 11:26:39 PM
Also, I want this second post, just in case.


Title: Re: File Parsing Boot Camp
Post by: Xiggah on January 23, 2012, 11:27:07 PM
And I'll take this last post too. No more needed.


Title: Re: File Parsing Boot Camp
Post by: DSX8 on January 23, 2012, 11:31:43 PM
ill add some stuff bout PAT0 files in the morning :P

mostly the header tho... im still going through the other sections in PAT0


Title: Re: File Parsing Boot Camp
Post by: Xiggah on January 23, 2012, 11:37:49 PM
Let's see..... we'll have this download link for the hex editor I use, HXD (http://mh-nexus.de/en/downloads.php?product=HxD), and some basic tutorial writing for now.

Thanks for the info Nymph, I could use some of those header informations. I'll probably have to use some pages from the Mario Kart Wiki page for now


Title: Re: File Parsing Boot Camp
Post by: DSX8 on January 23, 2012, 11:38:51 PM
no problem! i've like fixed alot of V4 PAT0 files and they were able to open file in BB once replaced with the unfixed file... so yeah :P

edit:
btw i've added u on msn... so ya hope u got my invite lol


Title: Re: File Parsing Boot Camp (Tutorial in the making)
Post by: DSX8 on January 24, 2012, 12:35:23 AM
this post is reserved by me


SCN0 Discussion:
Stage lighting, fog, and camera for stages! More soon to come

Header for brawl SCN0's:
Quote
Version# Offset: 00x8
Frame Offset:
Loop Offset:
---------------------------------------------------------------------------------------------------
PAT0 Discussion:
(PATtern) Texture pattern, an animation file that switches textures to simulate an animated texture.

Header for brawl V3 PAT0's:
Quote
Version# Offset: 00x8
Frame Offset:
How many sub-files offset:
Loop Offset:


Header for TvC (and other wii games) V4 PAT0's:
Quote
Version# Offset: 00x8
Frame Offset:
How many sub-files offset:
Loop Offset:

---------------------------------------------------------------------------------------------------
From Pikazz helpful Info:
Quote from: pikazz
okay, here we goes:


Header:
0x00 (4) PAT0
0x04 (4) String Offset Table
0x08 (4) Version
0x0C (4) Unknown1
0x10 (4) Pipeline offset
0x14 (4) "Texture Reference" (will describe this later)
0x18 (4) Unknown2, one offset that leading to a emtpy space
0x1C (4) Unknown3, often the same as Unknown2
0x20 (4) String Offset Table (Copy)
0x24 (4) Name of animation Offset
0x28 (4) Unknown4
0x2C (2) Frames
0x2E (2) NumEntries/Materials
0x30 (2) How many Textures (String Entries1)
0x32 (2) String Entries 2, often set as "00"
0x34 (4) Unknown5

PipeLine:
0x00 (4) How long PipeLine is
0x04 (4) How many Entries/IDs
0x08-1C IDs thingy
dont exactly know how the ID's working but this is importiant:
0x20 (4) It's name in String Offset (Pipelines offset + this offset = string offset)
0x24 (4) "link" offset (Pipeline offset + this offset = "Link" offset)
and the ID and Stirng/pattern offsets repeat on each entries

"Link" Offset:
0x00 (4) String Offset, linked to Materials name
0x04 (2) What Texture it will using if Flag is set on "7". if the flag is "5" this is set on "00"
0x06 (2) Flag: 5 using Pattern, 7 is Frozen
0x08 (4) Pattern Animation Offset (Link Offset + this = Pattern Offset). this is set on "00" if the Flag is set on "7"

Pattern Animation offset
0x00 (2) Unknown, possibly flag
0x02 (2) Unknown
0x04 (4)  unknown, Possible many flags in "bits"
0x08 (4) unknown, always(?) set on 00
0x0C (2) Which Texture will be shown
0x0E (2) Unknown, always(?) set on 00
0x10 (2) Which frame and possible with a flag with it (if it saying 41B8, the 4 is a flag and 1B8 is frame)

Repeat from 0x0C to 0x10 on how many frames

"Texture refserence" Offset
here is special and hard to explain, the textures here comes in order thanks by the "link" offset and it's the order the Pattern Animation in their "texture"

0x00 (4) Offset to their texture name in String Offset

String Table Offset:
well, all the names are in here, always in the end of the PAT0s

This is my work and I have done this in more than a year :srs:


Title: Re: File Parsing Boot Camp (Tutorial in the making)
Post by: deadname mcredactedface on January 24, 2012, 01:48:43 AM
Posting to subscribe.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Xiggah on January 24, 2012, 02:37:41 AM
There's a tutorial for hexing, I'll write one for file structures next.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: pikazz on January 24, 2012, 02:57:09 AM
PAT0 is already complete by me  :>.>palm: *looking for my files*
when a "nice looking form" by E_Y

EDIT: dont tell me that Smashboard deleted one section ;_;


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DSX8 on January 24, 2012, 03:01:49 AM
PAT0 is already complete by me  :>.>palm: *looking for my files*

EDIT: dont tell me that Smashboard deleted one section ;_;
:srs:
dont tell me u dont keep a backup saved in a .txt file :notimp:


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: pikazz on January 24, 2012, 03:07:10 AM
:srs: dont tell me u dont keep a backup saved in a .txt file :notimp:
nope cause it was over one year old :oshi: cause it was on smashboard and I never though they would delete that section but I can redo it cause I still remember 70-75% of it without using an PAT0 file as a reference D:


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DSX8 on January 24, 2012, 03:11:40 AM
nope cause it was over one year old :oshi: cause it was on smashboard and I never though they would delete that section but I can redo it cause I still remember 70-75% of it without using an PAT0 file as a reference D:
hmm.... well give me wat u kno... and ill add it then, but with major credit to u :P


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DarkPikachu on January 24, 2012, 03:57:44 AM
I've joined your camp simply by being cool :vyse:

all I can contribute to here is my REFF template for HexEdit 4.0 =3=
(I'm on wii so I can't do anything other than tell you about it) X(

EDIT:
I could use some help in the Melee area >_>
(Me and Milun can't do it all)

you can DL UMC 2.6d and my last (crappy) HexEdit 4.0 template (it works) if your interested in learning the files
NOTE: everything in my template needs to be renamed...

I think I have a download over at Brawl Imports...
if not, then my src is on gist.github.com
(search for 'DAT Template' on google and look for the gist link)

EDIT2:
btw, I only keep backups of my informative format posts:
UMC TMP format
MDL0 format (v9 and v11 (soon to merge))
Melee DAT format (needs an update BADLY)
others that I can't quite rmbr atm :P


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: ForOhFor Error on January 24, 2012, 04:16:08 AM
I might join. Also, there's a whole bunch of SSE filetypes to sift through.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DSX8 on January 24, 2012, 04:30:39 AM
lol TCLL... this is only for helping BJ with PAT0, REFF, the Physics from Havok, and SHP0 data... lol

but since u have data on REFF... sure give us a hand on that smexy data ;)

and welcome aboard 404


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DarkPikachu on January 24, 2012, 04:38:35 AM
Homer: ooooooooooohh why can't I share the lime light with Xiggah X(
lol


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: pikazz on January 24, 2012, 04:55:35 AM
okay, here we goes:

Header:
0x00 (4) PAT0
0x04 (4) String Offset Table
0x08 (4) Version
0x0C (4) Unknown1
0x10 (4) Pipeline offset
0x14 (4) "Texture Reference" (will describe this later)
0x18 (4) Unknown2, one offset that leading to a emtpy space
0x1C (4) Unknown3, often the same as Unknown2
0x20 (4) String Offset Table (Copy)
0x24 (4) Name of animation Offset
0x28 (4) Unknown4
0x2C (2) Frames
0x2E (2) NumEntries/Materials
0x30 (2) How many Textures (String Entries1)
0x32 (2) String Entries 2, often set as "00"
0x34 (4) Unknown5

PipeLine:
0x00 (4) How long PipeLine is
0x04 (4) How many Entries/IDs
0x08-1C IDs thingy
dont exactly know how the ID's working but this is importiant:
0x20 (4) It's name in String Offset (Pipelines offset + this offset = string offset)
0x24 (4) "link" offset (Pipeline offset + this offset = "Link" offset)
and the ID and Stirng/pattern offsets repeat on each entries

"Link" Offset:
0x00 (4) String Offset, linked to Materials name
0x04 (2) What Texture it will using if Flag is set on "7". if the flag is "5" this is set on "00"
0x06 (2) Flag: 5 using Pattern, 7 is Frozen
0x08 (4) Pattern Animation Offset (Link Offset + this = Pattern Offset). this is set on "00" if the Flag is set on "7"

Pattern Animation offset
0x00 (2) Unknown, possibly flag
0x02 (2) Unknown
0x04 (4)  unknown, Possible many flags in "bits"
0x08 (4) unknown, always(?) set on 00
0x0C (2) Which Texture will be shown
0x0E (2) Unknown, always(?) set on 00
0x10 (2) Which frame and possible with a flag with it (if it saying 41B8, the 4 is a flag and 1B8 is frame)

Repeat from 0x0C to 0x10 on how many frames

"Texture refserence" Offset
here is special and hard to explain, the textures here comes in order thanks by the "link" offset and it's the order the Pattern Animation in their "texture"

0x00 (4) Offset to their texture name in String Offset

String Table Offset:
well, all the names are in here, always in the end of the PAT0s

This is my work and I have done this in more than a year :srs:


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Sky Grounder on January 24, 2012, 06:23:41 AM
I'm aboard, though I will mostly help with stage/SSE files.

SCLA:
All stages have a SCLA file, most have an index of MiscData[30].
Here's the ones that have a different index:

STGJUNGLE.pac       MiscData[100]
STGPICTCHAT_xx.pac  MiscData[8]
STGDXBIGBLUE.pac    MiscData[70]
STGEDIT_0.pac       MiscData[903]
STGEDIT_1.pac       MiscData[1903]
STGEDIT_2.pac       MiscData[2903]
STGTARGETLV1.pac    MiscData[50]
STGTARGETLV2.pac    MiscData[50]
STGTARGETLv3.pac    MiscData[50]
STGTARGETLv4.pac    MiscData[50]
STGTARGETLv5.pac    MiscData[50]

*STGCHARAROLL_xx.pac doesn't have any.

There's also a SCLA in adventure_common_en.pac (MiscData[0]).

-----

The size of all SCLA files are 2832 (0xB10) bytes.

-----

Format:

Header: 0x0F (16 bytes)

0x00(4) - SCLA
0x04(4) - # of Entries (Always 0x20 (32))
0x08(4) - Nulls?
0x0C(4) - Nulls?

Offsets (based on # of Entries)
The offsets come right after the header.
Each offset is 4 bytes.
4 * 32 = 128 bytes (0x80)

After the offsets is the entries.
Each offset is 0x54 (84) bytes long.

I still haven't tried to find out what the values
in the entries do though.

BLOC:
BLOC is a file container (like BRRES) that contain other files.
But way more simple.
It appears in SSE files.

Format:

0x00(4) - BLOC
0x04(4) - # of entries
0x08(4) - Unknown
0x0C(4) - Nulls?

Offsets (starts at 0x10)
4 bytes * # of entries = size of string table


Formats BLOC can contain: GDOR GLOK GET1 GFSR GMOV GMOT GFT2 GENC GEPT GITM etc.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Spex130 on January 24, 2012, 09:29:52 AM
Holy crap. This thread actually went and happened. Wonderful!

I still need to know how I can help, though.

Edit: Ah. Nevermind. Found the tutorial.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Xiggah on January 24, 2012, 01:22:30 PM
Holy crap. This thread actually went and happened. Wonderful!

I still need to know how I can help, though.

Edit: Ah. Nevermind. Found the tutorial.

Don't worry, I'll up and make more tutorials as soon as I find the best way to identify structures myself... gotta find those old posts explaining these things.

Also, I'll try to include brief explanations of each file type so people can easily take an interest to different filetypes.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DSX8 on January 24, 2012, 02:14:32 PM
hmm wat else could i take a good look into.... i guess ill look back into SCN0 data


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Xiggah on January 24, 2012, 02:18:31 PM
SCN0 data is a good example. We should put together all of the undefined files that have looked into so far. Now where did all those file definitions run off too...

EDIT: Skygrounder, where did you find those file types anyways?


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Sky Grounder on January 24, 2012, 03:55:22 PM
SCN0 data is a good example. We should put together all of the undefined files that have looked into so far. Now where did all those file definitions run off too...

EDIT: Skygrounder, where did you find those file types anyways?
The SCLA file is found within every stage file, it's just never been talked about. (I don't know what purpose it serves though.)
As for the BLOC, it's found within SSE stages, and contain formats such as the ones I listed in the previous post (which might have something to do with auto-movement, doors, itemboxes, hitboxes, etc.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: BlackJax96 on January 24, 2012, 03:55:53 PM
I guess you could say I'm a part of this by default.
I'm just gonna post my stuff in C# code... no point reformatting it.

Btw, please don't call the Resource Group "Pipelines" >.>
Really, "relocation table" or "resource group" is a much better name for it...
We have to use like terms.

PAT0 parsing and rebuilding is done.
The only thing that needs to be fully figured out is the secondary flags. :srs:
I need to update Bero's code so it works better.

Anway, SHP0:
unsafe struct SHP0
    {
        public BRESCommonHeader _header;
        public bint _dataOffset;
        public bint _stringListOffset; //List of vertex node strings
        public bint _stringOffset;
        public bint _unk1; //0
        public bshort _numFrames;
        public bshort _numEntries;
        public bint _unk2; //0x00, 0x01

unsafe struct SHP0Entry
    {
        public bint _numEntries; //entries + 1 index part
        public bint _stringOffset;
        public bint _numIndices;
        public bint _unk;
        public bint _indiciesOffset;

I'm pretty sure the keyframes are I12 Keyframe Entries except with no frame scale.

    public unsafe struct I12Header
    {
        public bushort _numFrames;
        public bushort _unk;
        public bfloat _frameScale; //This is the value that isn't present in SHP0

public unsafe struct I12Entry
    {
        public bfloat _index;
        public bfloat _value;
        public bfloat _tangent;

Also, REFF definitely controls REFT images. It's not SRT0 but REFF that makes some of the images go together like a GIF. (like animated fire or a spinning coin)

Havok Physics:
public unsafe struct PhysicsHeader
    {
        public const uint Tag1 = 0x57E0E057;
        public const uint Tag2 = 0x10C0C010;

        public uint _tag1; //0x57E0E057
        public uint _tag2; //0x10C0C010
        public bint _unk2; //0
        public bint _unk3; //4

        public byte _unk4; //4
        public bshort _unk5; //1
        public byte _unk6; //1
        public bint _unk7; //3
        public bint _unk8; //1
        public bint _unk9; //0

        public bint _unk10; //0
        public buint _unk11; //Size/Offset

        //Three sections of offsets:
        //__classnames__
        //__data__
        //__types__

        public OffsetSection ClassNames;
        public OffsetSection Data;
        public OffsetSection Types;

        public String Name { get { return new String((sbyte*)Address + 0x28); } }
        
        public VoidPtr ClassNamesData { get { return Address + *(buint*)(Address + 0x54); } }
        public VoidPtr DataData { get { return Address + *(buint*)(Address + 0x84); } }
        public VoidPtr TypesData { get { return Address + *(buint*)(Address + 0xB4); } }
}

public unsafe struct OffsetSection
    {
        public fixed byte name[0x10];

        public bint _unk0; //-1
        public buint _dataOffset; //Main header is the base
        
        //Offsets to indices struct. _dataOffset is the base for everything.
        public buint _dataOffset1;
        public buint _dataOffset2;
        public buint _dataOffset3;
        public buint _dataOffset4;
        public buint _dataOffset5;
        public buint _dataOffset6;

        //When offsets begin to repeat, stop reading
        //Indices count is ((next data offset) - (current data offset)) / 2
        //Indices are padded to 0x10 with 0xFF

        public String Name { get { return new String((sbyte*)Address); } }
    }

public unsafe struct IndexGroup
    {
        //Base is the offset sections's _dataOffset
        //Still not sure exactly how these are used, but they're definitely a datalength or offset
        
        public buint _dataOffset; //Not necessarily...
        public buint _stringOffset; //Not necessarily...

Also, SCN0 is almost rebuildable. I finished STPM too.
SCN0 definitely needs more research, especially since it has a frame count. :srs:


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Sky Grounder on January 24, 2012, 04:06:49 PM
While we're talking about SCN0 + frames, I tried to port a SSE stage to final destination some time ago, and the result of the SceneData was that it kept shifting between light and dark.
I *think* this was the file:
http://www.mediafire.com/?6e0nwrebfid5hv2


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Spex130 on January 24, 2012, 04:18:56 PM
Does that animation count have anything to do with the lighting system on Battlefield?


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DSX8 on January 24, 2012, 04:23:01 PM
Does that animation count have anything to do with the lighting system on Battlefield?
yes it does... battlefields lighting has a framecount of 10800


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: Xiggah on January 24, 2012, 04:42:03 PM
yes it does... battlefields lighting has a framecount of 10800
Framecount of 10800, going off of memory here isn't that the same framecount for Battlefields stage animation? Seems like a typical light animation to me.

Anyways, I'll be adding the file structs to the first post.


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: SonicBrawler on January 24, 2012, 04:48:34 PM
I will keep an eye on this...


Title: Re: - — File Parsing Boot Camp — - (Intro to hexing tutorial up!)
Post by: DSX8 on January 24, 2012, 04:55:51 PM
Framecount of 10800, going off of memory here isn't that the same framecount for Battlefields stage animation? Seems like a typical light animation to me.

Anyways, I'll be adding the file structs to the first post.
yeah, u r indeed correct, but the light animation is somehow worked by the LightSet data... i really havent ran into frames in those files yet tho... still looking tho

but also... i did do a test on that framecount 2 weeks ago, u cut it by half and yeah the lighting did change for the loop


Title: Re: - — File Parsing Boot Camp — -
Post by: DarkPikachu on January 24, 2012, 05:57:20 PM
there's not much known about REFF files other than their structure is the exact same as REFT files >_>

but BJ is more able to post about it and knows about as much as me <_<

I just have HexEdit templates released on Brawl Imports for what I've done...
(my MDL0 template has an update that's pending)


Title: Re: - — File Parsing Boot Camp — -
Post by: Xiggah on January 24, 2012, 06:55:21 PM
there's not much known about REFF files other than their structure is the exact same as REFT files >_>

but BJ is more able to post about it and knows about as much as me <_<

I just have HexEdit templates released on Brawl Imports for what I've done...
(my MDL0 template has an update that's pending)

BJ came by and in the midst of his file structures said REFF files contain REFT images. I've seen REFFs with REFT images or holding AI data in FitCharacterMotionETCs, effect.pacs, boss files, etc.

As far as these structures go, I can go into Brawlbox and post the file structures for all of the defined file types, but I need to learn how to read them well first...


Title: Re: - — File Parsing Boot Camp — -
Post by: DarkPikachu on January 24, 2012, 07:01:15 PM
O.O interesting

I'll have to take a look at that myself...

maybe that's what the 32bit value at the beginning of the data means >_>


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 25, 2012, 01:36:24 AM
O.O interesting

I'll have to take a look at that myself...

maybe that's what the 32bit value at the beginning of the data means >_>

Please do! I want to encourage people to hex and look into these files in this thread, and you're very good at that.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 02:20:03 AM
I am looking in REFF right now :3

I believe I see something instresting!

but I would like the REFT header first since it's already discover if I am not wrong o.o


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 25, 2012, 02:30:25 AM
I am looking in REFF right now :3

I believe I see something instresting!

but I would like the REFT header first since it's already discover if I am not wrong o.o
wait till BJ is back from his awesome day at skewl... he loves skewl so much that it kills him on the inside daily :vyse:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Sky Grounder on January 25, 2012, 02:32:32 AM
wait till BJ is back from his awesome day at skewl... he loves skewl so much that it kills him on the inside daily :vyse:
If I'm guessing right, it's about 4:30 AM at his place right now...


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 25, 2012, 02:33:20 AM
If I'm guessing right, it's about 4:30 AM at his place right now...
nope... he's the same time as me.. 1:30am


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 25, 2012, 02:34:59 AM
We can just count the hours until BlackJax gets back, can't we.... or we could surprise him and find out the REFT headers ourselves :vyse:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 25, 2012, 02:36:34 AM
We can just count the hours until BlackJax gets back, can't we.... or we could surprise him and find out the REFT headers ourselves :vyse:
BJ already has REFT... he just needs REFF to understand how the REFT works.. xD


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 02:38:34 AM
We can just count the hours until BlackJax gets back, can't we.... or we could surprise him and find out the REFT headers ourselves :vyse:
well, I can't wait and have nothing to do right now so... *goes away to see about the REFF/REFT header by my own* :vyse:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 25, 2012, 02:44:22 AM
well, I can't wait and have nothing to do right now so... *goes away to see about the REFF/REFT header by my own* :vyse:

That's the spirit!... but enough of that, is this link (http://brawlimports.proboards.com/index.cgi?board=templates&action=display&thread=9) any relevant? It comes from TCLL's programming forum, seems to have some useful data and links on REFT files.

EDIT: I might have the REFT data? :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 02:57:14 AM
busted at 60-70% on my own :vyse:
will write it soon once it's 100%
That's the spirit!... but enough of that, is this link ([url]http://brawlimports.proboards.com/index.cgi?board=templates&action=display&thread=9[/url]) any relevant? It comes from TCLL's programming forum, seems to have some useful data and links on REFT files. EDIT: I might have the REFT data? :P

lol, I didn't saw that site O.o


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 25, 2012, 03:02:22 AM
busted at 60-70% on my own :vyse:
will write it soon once it's 100%lol, I didn't saw that site O.o

I hope it helps. It even claims to have a link to a REFT viewer, apparantely it's a common image file.

But if you're at a dead end and you're bored, you could always help out here.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 03:40:43 AM
okay, here we go with the header of the REFF file

First Header:
0x00 (4) REFF, the type file
0x04 (2) always FE FF,  unknown. possible Fileidex/id ect
0x06 (2) unknown, always 00 07
0x08 (4) How big the file is
0x0C (2) unknown, think it's a jump to "next REFF header"
0x0E (2) possible ID

Second Header:
0x00 (4) REFF, the type file again!
0x04 (4) how big the file is again but 18 shorter
0x08 (4) offset to the special "String Table Link section!" or how big this header is, often set as 18.
0x0C (4) unknown
0x10 (4) unknown
0x14 (2) unknown, always set on 6

String Table Link Section:
this is hard to write since it's pretty messed up D:
first it's the REFF name example "purin" or "yoshi",
after the name the "String Table Link Section" length and after that it's how many entries it is in.

then it's using that follow on the rest of the REFF part/names ect.
First it's the "sections" name example "PtcPurinKorogaru", after that it's the offset (This offset + the beginning of STLS (30) = the offset!)
and after it its how big the file are (Offset before + this offset  = how big that file is)

and repeat on every "file"/entries




Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 25, 2012, 03:42:16 AM
okay, here we go with the header of the REFF file

First Header:
0x00 (4) REFF, the type file
0x04 (2) always FE FF,  unknown. possible Fileidex/id ect
0x06 (2) unknown, always 00 07
0x08 (4) How big the file is
0x0C (2) unknown, think it's a jump to "next REFF header"
0x0E (2) possible ID

Second Header:
0x00 (4) REFF, the type file again!
0x04 (4) how big the file is again but 18 shorter
0x08 (4) offset to the special "String Table Link section!" or how big this header is, often set as 18.
0x0C (4) unknown
0x10 (4) unknown
0x14 (2) unknown, always set on 6

String Table Link Section:
this is hard to write since it's pretty messed up  :kdance:
first it's the REFF name example "purin" or "yoshi",
after the name the "String Table Link Section" length and after that it's how many entries it is in.

then it's using that follow on the rest of the REFF part/names ect.
First it's the "sections" name example "PtcPurinKorogaru", after that it's the offset (This offset + the beginning of STLS (28) = the offset!)
and after it its how big the file are (Offset before + this offset  = how big that file is)

and repeat on every "file"/entries

hmm... but how exactly does it work with the REFT files?


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 03:46:45 AM
hmm... but how exactly does it work with the REFT files?
cause REFT using the "same" method in the header, I looked between those 2 and saw no diffirens xD

but exactly how they are "connected" are still a mistery, I believe it's EFLS file that connecting those 2 together or REFF holding it in deeper :/


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 25, 2012, 03:48:20 AM
cause REFT using the "same" method in the header, I looked between those 2 and saw no diffirens xD

but exactly how they are "connected" are still a mistery, I believe it's EFLS file that connecting those 2 together or REFF holding it in deeper :/
well u did say its a big mess.... hmm :srs:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 04:06:43 AM
well u did say its a big mess.... hmm :srs:

this is what I meant:
(http://i42.tinypic.com/124fnn5.png)
Blackbox: REFFs name
Greenbox: how big the "entries" is in STLS
Bluebox: How many entries

Redbox: the REFF sections name
purplebox: Offset to it
pinkbox: how big the REFF section/entrie is

and repeat on each entrie with Redbox, PurpleBox and Pinkbox

EDIT: I found out how the REFF is connected with each other :happy: but I haven't found out the offset or something like that lol.
I am still cuseious houw REFF and REFT are connected

EDIT2: allI know how the REFF and REFT are connected is that REFF entrie having the REFT name in it if it using them :srs:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 25, 2012, 04:39:18 AM
you know...
I do have an REFT template released on Brawl Imports...
(I think it needs a small update though)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 25, 2012, 04:43:33 AM
cause REFT using the "same" method in the header, I looked between those 2 and saw no diffirens xD

but exactly how they are "connected" are still a mistery, I believe it's EFLS file that connecting those 2 together or REFF holding it in deeper :/
sry for D-Post...

all I can say is I've seen REFT image names in the REFF file...

I've also noticed a small hex patteren in one of the datas...
(I'll have to find it again) X(
but it follows a count exactly


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 04:45:01 AM
you know... I do have an REFT template released on Brawl Imports... (I think it needs a small update though)
I know but it wasn't wrote like this I did.
also, I started before Xiggah gave me the link to it D:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 25, 2012, 04:50:31 AM
btw...

I found out what the '0xFEFF' ment...

FEFF = little endian
FFFF = big endian


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 25, 2012, 05:27:52 AM
btw... I found out what the '0xFEFF' ment... FEFF = little endian FFFF = big endian
that's good :3

update: I found out that "Unk6" in the REFFs entries is frames!
I am 95% sure!


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Eternal Yoshi on January 25, 2012, 04:58:15 PM
I'll join the party so I can complete the Event Match and Classic mode formats.

Also stuff from MGS2.
Event Match format
Code:
Header Size - 0x50

0x08 = Match Type(Value is at 2 for Coin event match)
0x0C = Time Limit
0x1F = Stage ID
0x38 = Game Speed
0x3C = Camera Shaking control
0x46 = Music ID

PlayerData Size = 0x38
---------------------------------
0x00  = byte CharacterID
0x01  = byte Status (Normal,Metal,Invisible,BEEP?)
0x04  = float Size(Scaling)
0x08  = Team Flag
0x0C  = ?????(Easy)
0x0F  = Offense Ratio(Easy)
0x11  = Defense Ratio(Easy)
0x14  = Stock count(Easy)
0x18  = Starting Damage(Easy)
0x1B  = ?????(Normal)
0x1D  = Offense Ratio(Normal)
0x1F  = Defense Ratio(Normal)
0x22  = Stock count(Normal)
0x26  = Starting Damage(Normal)
0x28  = ?????(Hard)
0x2B  = Offense Ratio(Hard)
0x2D  = Defense Ratio(Hard)
0x30  = Stock count(Hard)
0x34  = Starting Damage(Hard)

http://www.smashboards.com/showthread.php?t=288062


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: BlackJax96 on January 25, 2012, 05:04:45 PM
You guys... REFT and REFF rebuilding is COMPLETE.
All I need figured out is the actual DATA contained in the REFF. Not the string relocation offsets. Just the data.
REFT is 100% rebuildable, but the texture importer needs a few tweaks to work right with it. I've replaced Peach's hearts with stars as a test and it looked fine in BB, but they appeared swizzled in-game.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Eternal Yoshi on January 25, 2012, 05:23:53 PM
That's good to know. Now we can focus on other things, like SHP0 and the event match files.

For the most part, All that remains for the event matches is part of the header which dictates success and failure conditions.

For those who don't have a memo, You can edit Event Matches in MiscData[6] via common2.pac ONLY WITH TABUU.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 25, 2012, 06:31:44 PM
Event Matches sounds like a nice thing to edit. I think I could find a few nice things in Event files if I get any good at file parsing.

Oh, and would anyone be interested if I posted the file structures of files already defined from Brawlbox? BJ made them nice and readable in the source code.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 25, 2012, 07:54:59 PM
Event Matches sounds like a nice thing to edit. I think I could find a few nice things in Event files if I get any good at file parsing.

Oh, and would anyone be interested if I posted the file structures of files already defined from Brawlbox? BJ made them nice and readable in the source code.
go for it ;D

actually, it'd be real helpful :)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: pikazz on January 26, 2012, 05:27:52 AM
You guys... REFT and REFF rebuilding is COMPLETE. All I need figured out is the actual DATA contained in the REFF. Not the string relocation offsets. Just the data. REFT is 100% rebuildable, but the texture importer needs a few tweaks to work right with it. I've replaced Peach's hearts with stars as a test and it looked fine in BB, but they appeared swizzled in-game.
so you are not proud? D;

will do some research in the REFF about the data


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 09:58:32 AM
OMG D:
I hope I didn't release my last MDL0 template like this :|

I never noticed it because I havn't been able to revise it until now, but

1 - the vertex attributes don't work as the command var isn't referenced in the switch.
2 - the entire 2nd half of the VtxAttrHi is labled as UV_7 in the primitives

but, I was trying to be quick and release it, and the area had nothing on it, so...
at least now it's fixed
(and I sped it up a little (can only do a minor speedup))

the requirements of the MDL0 structure causes it to load extremely slow >_>

I've suggested a template buffer for HexEdit to Andrew though,
so he should hopefully implement that.
(would give a 10x speedup)

the template takes over an hour to load when loading Pachirisu (v11) in design mode D:
(about 15 minutes out of design mode)
and yes, I've fixed the v9 and v11 implementations, so that will be in my next release ;)

I'll disable alot so it won't take forever to load...
(best I can limit is 3-5 minutes for small files)

EDIT:
forgot to mention...
the registers (shaders and nodes), attributes (objects) and primitives (objects) are what drastically increase loading time...

the constant repeat in structures is what causes it >_>


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 12:26:14 PM
D-Post... IK... shoot me.

I've just UD'd BJ's thread with important info about BP registers.

IK my template is wrong, but I had the wrong idea...
I'm working on correcting it right now ;)

btw, I looked at my last template release...
it was before I've finished re-defining everything XDD
(my slowest release yet)

hopefully this next one will be alot faster than that :)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 01:23:16 PM
It's all right, share your progress. By the way, are there any specific defined structures you want me to post from Brawlbox?


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 01:40:03 PM
It's all right, share your progress. By the way, are there any specific defined structures you want me to post from Brawlbox?
just what I'm interested in :P

that being:
Resource Groups (still needed)
Materials (because of the shaders)
Nodes (somewhat still interested in these)
Objects (I'm sure there's been new findings)

that's it :)

btw...
I don't really have a beef with multi-posting...
-was flammed and warned for someone else's complaints... (I kept multi-posting)
*cough* NSM *cough cough*
I blame the admins for removing the "show signature" option...
I'd at least do that >:O

*sigh* and now I rmbr why I hate this forum -.-*
but I'm not gonna rant about it...
this is about file data,
not why people are P'd at forums


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 01:48:06 PM
All right, I'll go gather up some of that source code. By the way, do you know anything about nulling normals or similar things like that?


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 01:53:04 PM
if I'm right...
n = (0,0,0)*count

I think that's what does it >_>


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 02:08:45 PM

Resource Groups (still needed)
Materials (because of the shaders)
Nodes (somewhat still interested in these)
Objects (I'm sure there's been new findings)
Let's see if I can do this right.... I'm just posting the headers here, I don't know if you wanted me to lay any of it out or anything like that.

Resource Groups: Do you mean the node trees or the Definitions?

Materials:
        public bint _dataLen;
        public bint _mdl0Offset;
        public bint _stringOffset;
        public bint _index;
        public buint _isXLU; //0x00 0x80000000 for XLU textures
        public byte _flag1; //Texture count
        public byte _numLayers; //Not exactly
        public byte _flag3; //Repeated on shader
        public byte _flag4;
        public bint _cull; //0x02, XLU = 0
        public byte _flag5;
        public byte _flag6;
        public byte _flag7; //Texture layers
        public byte _flag8;
        public bint _unk3;
        public bint _unk4; // -1, 0xFFFFFFFF
        public bint _shaderOffset;
        public bint _numTextures;
        public bint _MatRefOffset; //1044, or 1048 for v11 MDL0
        public bint _part2Offset; //0
        public bint _dlOffset; //Offset to display list(s).
        public bint _unk6; //0


Nodes.... there are 5 different versions. Which ones do you want?


Objects
        public const uint Size = 0x64;

        public bint _totalLength;
        public bint _mdl0Offset;
        public bint _nodeId; //Single-bind node

        public CPVertexFormat _vertexFormat;
        public XFVertexSpecs _vertexSpecs;

        public bint _defSize; //Size of def block including padding? Always 0xE0?
        public bint _defFlags; //0x80, sometimes 0xA0
        public bint _defOffset; //Relative to defSize field

        public bint _dataLen1; //Size of primitives
        public bint _dataLen2; //Same as previous
        public bint _dataOffset; //Relative to dataLen1

        public XFArrayFlags _arrayFlags; //Used to enable element arrays?

        public bint _unk3; //0
        public bint _stringOffset;
        public bint _index;

        public bint _numVertices;
        public bint _numFaces;

        public bshort _vertexId;
        public bshort _normalId;
        public fixed short _colorIds[2];       
        public fixed short _uids[8];

        public bint _nodeTableOffset;


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 02:28:26 PM
ok umm...

I prbly should just call them Resource Lists since you keep getting confused over them XD

and Nodes == Material-Nodes
they're currently called Shaders in Brbx...
(the term needs correction) >_>

here's some useful info BJ doesn't seem to want to listen to: <_<

Shaders are located w/in the material and control things such as fur, grass, smoke, or water
they can handle multiple textures (up to 10 in blender)
and can also control things such as bump-maps, texture displacement, multi-texturing (with an alpha)

Material-Nodes are a seperate instance from the material, and control material color (of it's linked material), texture color/brightness, light effects and color, among other things I've yet to look into <_<

I have more info in my old MDL0 thread >_>

I've also animated Pikachu's eyes with a dynamic node :D
although you have to do double the work to animate it with the script <_<

yeh... I've still got a ways to go in game development yet -.-
but I'll get there ;)

NOTE:
I'm actually describing this stuff as it is in Blender...
(JIC people want to think I'm making this stuff up) >_>
^you have no idea how much I've been through Xiggah D:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 02:40:20 PM
ok umm...

I prbly should just call them Resource Lists since you keep getting confused over them XD

and Nodes == Material-Nodes
they're currently called Shaders in Brbx...
(the term needs correction) >_>

here's some useful info BJ doesn't seem to want to listen to: <_<

Shaders are located w/in the material and control things such as fur, grass, smoke, or water
they can handle multiple textures (up to 10 in blender)
and can also control things such as bump-maps, texture displacement, multi-texturing (with an alpha)

Material-Nodes are a seperate instance from the material, and control material color (of it's linked material), texture color/brightness, light effects and color, among other things I've yet to look into <_<

I have more info in my old MDL0 thread >_>

I've also animated Pikachu's eyes with a dynamic node :D
although you have to do double the work to animate it with the script <_<

yeh... I've still got a ways to go in game development yet -.-
but I'll get there ;)

NOTE:
I'm actually describing this stuff as it is in Blender...
(JIC people want to think I'm making this stuff up) >_>
^you have no idea how much I've been through Xiggah D:

Lots of work, that's what you've been through. Much more than I at any rate. Those materials and shaders, we need a nice guide on how to make nice effects with them. I'm specifically interested in decreasing the amount of lighting, I suppose that would be with the shader.

Shaders: There are also TEVblocks, do you want to see those?
        public const int Size = 32;

        public bint _dataLength; //Always 512
        public bint _mdl0Offset;
        public bint _index;
        public byte _flag; //Same as material
        public byte _res0, _res1, _res2; //Always 0. Reserved?
        public sbyte _ref0, _ref1, _ref2, _ref3, _ref4, _ref5, _ref6, _ref7;
        public int _pad0, _pad1; //Always 0

        //Display list
        public ShaderTevBlock* DisplayList { get { return (ShaderTevBlock*)(Address + Size); } }
        //There are 8 structures max following the display list, each 0x30 in length.
        //Each structure has 9 commands.


Definitions: I don't know if this is the right Resource Entry or not, but here they are. It mostly refers to other node types.
public byte _type;

        private VoidPtr Address { get { fixed (void* ptr = &this)return ptr; } }

        public MDL0NodeType2* Type2Data { get { return (MDL0NodeType2*)(Address + 1); } }
        public MDL0NodeType3* Type3Data { get { return (MDL0NodeType3*)(Address + 1); } }
        public MDL0NodeType4* Type4Data { get { return (MDL0NodeType4*)(Address + 1); } }
        public MDL0NodeType5* Type5Data { get { return (MDL0NodeType5*)(Address + 1); } }

        public MDL0DefEntry* Next
        {
            get
            {
                switch (_type)
                {
                    case 2: return (MDL0DefEntry*)(Type2Data + 1);
                    case 3: return (MDL0DefEntry*)(Type3Data + 1);
                    case 4: return (MDL0DefEntry*)(Type4Data + 1);
                    case 5: return (MDL0DefEntry*)(Type5Data + 1);


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 02:56:52 PM
here's a small example of rendering hair with shaders:
(http://hamsterking.com/blender/images/p-fig18.jpg)

decreasing the lighting??
are you talking about the specular part of the materials??
(I always use black to get something towards Pika's skin look)
^eg: my ava

although I think that can be controlled through nodes as well...

TBH, I really don't think actual materials are used anymore...
(I think it's all dynamically controlled through the shader/node)

and the resource lists are what the MDL0 header contains offsets to :P

but yur OK with the Defs being posted XD


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 03:13:09 PM
I don't know the difference between specular or diffuse or ambient yet... so no idea :P

Brawlbox's source code has BJ's name definitions, that's all. Should I still post those tidbits of code?


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 03:19:12 PM
I don't know the difference between specular or diffuse or ambient yet... so no idea :P

Brawlbox's source code has BJ's name definitions, that's all. Should I still post those tidbits of code?
yea, go ahead an post em :)

and for the colors:
diffuse is your main model color
specular is the light reflection color
ambiant *forgets how it works* :P
(ambiant light would be like light in a room (not reflected from a direct source))

EDIT:
ambient *correction* color is the reflected color from the object to another object


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 03:32:06 PM
yea, go ahead an post em :)

and for the colors:
diffuse is your main model color
specular is the light reflection color
ambiant *forgets how it works* :P
(ambiant light would be like light in a room (not reflected from a direct source))

I didn't even know Brawl had light reflection, I thought it was all direct light. Anyways, I might just have to toy around with the Shader settings until I find the right values. Too bad I have idea what any of it means :P

Here, have some.... node types. Here, have the entire code snippet. now complete without accidental italicization.


Code:
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe abstract class MDL0NodeClass
    {
        public static object Create(ref VoidPtr addr)
        {
            object n = null;
            switch (*(byte*)addr++)
            {
                case 2: { n = Marshal.PtrToStructure(addr, typeof(MDL0Node2Class)); addr += MDL0Node2Class.Size; break; }
                case 3: { n = new MDL0Node3Class((MDL0NodeType3*)addr); addr += ((MDL0Node3Class)n).GetSize(); break; }
                case 4: { n = Marshal.PtrToStructure(addr, typeof(MDL0NodeType4)); addr += MDL0NodeType4.Size; break; }
                case 5: { n = Marshal.PtrToStructure(addr, typeof(MDL0NodeType5)); addr += MDL0NodeType5.Size; break; }
            }
            return n;
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe class MDL0Node2Class : MDL0NodeClass
    {
        public const uint Size = 0x04;

        public bushort _boneIndex;
        public bushort _parentNodeIndex;

        public ushort BoneIndex { get { return _boneIndex; } set { _boneIndex = value; } }
        public ushort ParentNodeIndex { get { return _parentNodeIndex; } set { _parentNodeIndex = value; } }
       
        public override string ToString()
        {
            return string.Format("Type2 (Bone Index:{0}, Parent Node Index:{1})", BoneIndex, ParentNodeIndex);
        }
    }

    public unsafe class MDL0Node3Class
    {
        public bushort _id;
        public List<MDL0NodeType3Entry> _entries = new List<MDL0NodeType3Entry>();

        public unsafe MDL0Node3Class(MDL0NodeType3* ptr)
        {
            _id = ptr->_id;
            for (int i = 0; i < ptr->_numEntries; i++)
                _entries.Add(ptr->Entries[i]);
        }

        public ushort Id { get { return _id; } set { _id = value; } }
        public MDL0NodeType3Entry[] Entries { get { return _entries.ToArray(); } }

        public int GetSize() { return 3 + (_entries.Count * MDL0NodeType3Entry.Size); }

        public override string ToString()
        {
            return string.Format("Type3 (ID:{0})", Id);
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType2
    {
        public const int Size = 0x04;

        public bushort _index;
        public bushort _parentId;

        public ushort Index { get { return _index; } set { _index = value; } }
        public ushort ParentId { get { return _parentId; } set { _parentId = value; } }

        public override string ToString()
        {
            return string.Format("Type2 (Index:{0},ParentID:{1})", Index, ParentId);
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType3
    {
        public const int Size = 0x03;

        public bushort _id;
        public byte _numEntries;

        private VoidPtr Address { get { fixed (void* ptr = &this)return ptr; } }

        public MDL0NodeType3Entry* Entries { get { return (MDL0NodeType3Entry*)(Address + 3); } }

    }
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType3Entry
    {
        public const int Size = 0x06;

        public bushort _id;
        public bfloat _value;

        public ushort Id { get { return _id; } set { _id = value; } }
        public float Value { get { return _value; } set { _value = value; } }

        public override string ToString()
        {
            return String.Format("Type3Entry (ID:{0},Weight:{1})", Id, Value);
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType4
    {
        public const uint Size = 0x07;

        public bushort _materialIndex;
        public bushort _polygonIndex;
        public bushort _boneIndex;
        public byte _val4;

        public ushort MaterialId { get { return _materialIndex; } set { _materialIndex = value; } }
        public ushort PolygonId { get { return _polygonIndex; } set { _polygonIndex = value; } }
        public ushort BoneIndex { get { return _boneIndex; } set { _boneIndex = value; } }
        public byte Val4 { get { return _val4; } set { _val4 = value; } }

        public override string ToString()
        {
            return string.Format("Type4 (MatID:{0},PolyID:{1},BoneIndex:{2},Unk:{3})", MaterialId, PolygonId, BoneIndex, Val4);
        }
    }

    //Links node IDs with indexes
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType5
    {
        public const uint Size = 0x04;

        public bushort _id; //Node Id
        public bushort _index; //Node Index

        public int Id { get { return _id; } set { _id = (ushort)value; } }
        public int Index { get { return _index; } set { _index = (ushort)value; } }

        public override string ToString()
        {
            return string.Format("Type5 (ID:{0},Index:{1})", Id, Index);
        }
    }


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 03:42:37 PM
I didn't even know Brawl had light reflection, I thought it was all direct light. Anyways, I might just have to toy around with the Shader settings until I find the right values. Too bad I have idea what any of it means :P

Here, have some.... node types. Here, have the entire code snippet. now complete without accidental italicization.


Code:
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe abstract class MDL0NodeClass
    {
        public static object Create(ref VoidPtr addr)
        {
            object n = null;
            switch (*(byte*)addr++)
            {
                case 2: { n = Marshal.PtrToStructure(addr, typeof(MDL0Node2Class)); addr += MDL0Node2Class.Size; break; }
                case 3: { n = new MDL0Node3Class((MDL0NodeType3*)addr); addr += ((MDL0Node3Class)n).GetSize(); break; }
                case 4: { n = Marshal.PtrToStructure(addr, typeof(MDL0NodeType4)); addr += MDL0NodeType4.Size; break; }
                case 5: { n = Marshal.PtrToStructure(addr, typeof(MDL0NodeType5)); addr += MDL0NodeType5.Size; break; }
            }
            return n;
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe class MDL0Node2Class : MDL0NodeClass
    {
        public const uint Size = 0x04;

        public bushort _boneIndex;
        public bushort _parentNodeIndex;

        public ushort BoneIndex { get { return _boneIndex; } set { _boneIndex = value; } }
        public ushort ParentNodeIndex { get { return _parentNodeIndex; } set { _parentNodeIndex = value; } }
        
        public override string ToString()
        {
            return string.Format("Type2 (Bone Index:{0}, Parent Node Index:{1})", BoneIndex, ParentNodeIndex);
        }
    }

    public unsafe class MDL0Node3Class
    {
        public bushort _id;
        public List<MDL0NodeType3Entry> _entries = new List<MDL0NodeType3Entry>();

        public unsafe MDL0Node3Class(MDL0NodeType3* ptr)
        {
            _id = ptr->_id;
            for (int i = 0; i < ptr->_numEntries; i++)
                _entries.Add(ptr->Entries[i]);
        }

        public ushort Id { get { return _id; } set { _id = value; } }
        public MDL0NodeType3Entry[] Entries { get { return _entries.ToArray(); } }

        public int GetSize() { return 3 + (_entries.Count * MDL0NodeType3Entry.Size); }

        public override string ToString()
        {
            return string.Format("Type3 (ID:{0})", Id);
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType2
    {
        public const int Size = 0x04;

        public bushort _index;
        public bushort _parentId;

        public ushort Index { get { return _index; } set { _index = value; } }
        public ushort ParentId { get { return _parentId; } set { _parentId = value; } }

        public override string ToString()
        {
            return string.Format("Type2 (Index:{0},ParentID:{1})", Index, ParentId);
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType3
    {
        public const int Size = 0x03;

        public bushort _id;
        public byte _numEntries;

        private VoidPtr Address { get { fixed (void* ptr = &this)return ptr; } }

        public MDL0NodeType3Entry* Entries { get { return (MDL0NodeType3Entry*)(Address + 3); } }

    }
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType3Entry
    {
        public const int Size = 0x06;

        public bushort _id;
        public bfloat _value;

        public ushort Id { get { return _id; } set { _id = value; } }
        public float Value { get { return _value; } set { _value = value; } }

        public override string ToString()
        {
            return String.Format("Type3Entry (ID:{0},Weight:{1})", Id, Value);
        }
    }

    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType4
    {
        public const uint Size = 0x07;

        public bushort _materialIndex;
        public bushort _polygonIndex;
        public bushort _boneIndex;
        public byte _val4;

        public ushort MaterialId { get { return _materialIndex; } set { _materialIndex = value; } }
        public ushort PolygonId { get { return _polygonIndex; } set { _polygonIndex = value; } }
        public ushort BoneIndex { get { return _boneIndex; } set { _boneIndex = value; } }
        public byte Val4 { get { return _val4; } set { _val4 = value; } }

        public override string ToString()
        {
            return string.Format("Type4 (MatID:{0},PolyID:{1},BoneIndex:{2},Unk:{3})", MaterialId, PolygonId, BoneIndex, Val4);
        }
    }

    //Links node IDs with indexes
    [StructLayout(LayoutKind.Sequential, Pack = 1)]
    public unsafe struct MDL0NodeType5
    {
        public const uint Size = 0x04;

        public bushort _id; //Node Id
        public bushort _index; //Node Index

        public int Id { get { return _id; } set { _id = (ushort)value; } }
        public int Index { get { return _index; } set { _index = (ushort)value; } }

        public override string ToString()
        {
            return string.Format("Type5 (ID:{0},Index:{1})", Id, Index);
        }
    }
all I can say is google image search: 'specular color'
you should get the idea right off ;)
(it's the lightest color)

EDIT:
Samus has a specular on top of a texture reflection


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 03:45:20 PM
All right. Just let me know if you need anything else, I'll be researching shaders in the meantime.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 03:47:50 PM
you never posted the Resourse Groups
or as I call them, Relocation Groups XDD

and go for it my friend :D


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 26, 2012, 03:56:05 PM
all I can say is google image search: 'specular color'
you should get the idea right off ;)
(it's the lightest color)

EDIT:
Samus has a specular on top of a texture reflection
yeah and that follows the camera too :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 04:18:55 PM
yeah and that follows the camera too :P
Note: I've never seen Samus's data...
I'm only getting my info from the screen :P

but I can replicate the effects of a model if I have an OBJ with just textures :af:
(I could replicate Samus if I had the bare model (no materials, textures, or anything visual))
^well, maybe not 100% as I don't know every trick just yet :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 04:30:03 PM
There isn't much of a need to recreate something that's already made... I could import Samus's model much faster than you could recreate it :vyse:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 05:32:23 PM
There isn't much of a need to recreate something that's already made... I could import Samus's model much faster than you could recreate it :vyse:
well why the F not XD [/meme]

wouldn't you want to test your own skills :D


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 05:38:48 PM
well why the F not XD [/meme]

wouldn't you want to test your own skills :D
I do, but I often lose myself in doing something by hand that already gets done quickly by a computer. That's why I'd rather hex the module files, something that hasn't been done yet, rather than do something that already has been done. As a matter of fact, I would LOVE to hex the module files. First thing to do after I learn how to read file structures.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: BlackJax96 on January 26, 2012, 06:10:33 PM
All right. Just let me know if you need anything else, I'll be researching shaders in the meantime.

Don't focus on researching shaders, I already know exactly how they work. I'll be making the enums for shaders and shader stages much easier to understand, and then I'll be finishing the shader explanation on my thread. It's actually quite easy to understand, once all the data is parsed and in one location (which it is), along with the SDK's explanation. Plus, I think I've figured out a way to render them thanks to the Dolphin Emu svn.

I would really recommend looking into SHP0. It's really not a big file to parse, but somebody just needs to go do it like I did with SRT0.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 07:14:17 PM
Don't focus on researching shaders, I already know exactly how they work. I'll be making the enums for shaders and shader stages much easier to understand, and then I'll be finishing the shader explanation on my thread. It's actually quite easy to understand, once all the data is parsed and in one location (which it is), along with the SDK's explanation. Plus, I think I've figured out a way to render them thanks to the Dolphin Emu svn.

I would really recommend looking into SHP0. It's really not a big file to parse, but somebody just needs to go do it like I did with SRT0.
I hope you know we're talking about actual shaders...
(the ones that render the hair in the pic I posted earlier)
so I don't think you know how shaders work :P

sry bout getting off on a rant like that...
but dude... those aren't shaders... plain and simple

and that's comeing from the only one in the community who's actually into game development :srs:
(that's the reason I'm a little uppy atm)

why don't you look into rendering fur and what not BJ :)
(what I've described below)

EDIT:
and Xiggah...
it would really be of some help looking into the game scripts (REL files)
I've been picking them up every once in a while,
but never really got anywhere due to other stuff and projects and stuff :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: BlackJax96 on January 26, 2012, 07:23:00 PM
I hope you know we're talking about actual shaders...
(the ones that render the hair in the pic I posted earlier)
so I don't think you know how shaders work :P

sry bout getting off on a rant like that...
but dude... those aren't shaders... plain and simple

and that's comeing from the only one in the community who's actually into game development :srs:
(that's the reason I'm a little uppy atm)

why don't you look into rendering fur and what not BJ :)
(what I've described below)

Brawl can't render fur with only a shader. All it does is manipulate the final color fragment for each vertex based on your input.

Nintendo's shaders != Blender's "material nodes".

You'd have to create a an object with multiple layers to it and apply a texture of fur to it, which would basically simulate fur (like Fox, for example), then use Alpha Function to allow transparency. (Alpha Function is used for full transparent pixels, while Blend is used for opaque pixels. They can't be used at the same time) Really, I know exactly how shaders work. Sure you can deny what I'm saying right now, but you can't deny me once I render them in Brawlbox's previewer exactly how they display in-game. :srs:

The first "TEV Block" in the material sets the Color Inputs for the shader. The second "TEV Block" in the material sets the Konstant Colors for the shader. The first "TEV Block" in the shader itself is actually the Konstant Selection Swap Table.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 07:39:37 PM
BJ...
I'm not denying what you say... ok...

why I call them nodes:
Pika's EyeYellow node basically just adjusts the brightness of a texture, and then applys the result based on an alpha to the original
(I'm not exactly sure what the color values do in it though) <_<
the main node simply defines the model's color (I think)
^havn't looked into any other one but EyeYellow

anyways...
what I'm trying to say is anything in those structures is entirely compatible with Blenders Material nodes
(they can be replicated entirely)

the texture layers are defined in the material's shader (both in brawl and blender)
although the XF data continues to confuse me >_>

and I'm talking about actual fur...
not that fake crap of using layered models with the same texture XDD

the wii is capable (I do believe) of those kind of graphics
it just needs to be figured out <_<

and I know I'm sounding low to you right now...
I'm just tired of running into dead ends everywhere I go... -.-
(I'm not as in-depth into it as you are)
^but I know what the data does (in basic terms)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: BlackJax96 on January 26, 2012, 07:43:24 PM
the wii is capable (I do believe) of those kind of graphics
it just needs to be figured out <_<

and I know I'm sounding low to you right now...
I'm just tired of running into dead ends everywhere I go... -.-

It's possible to do, just not with only a shader.
And maybe you wouldn't run into dead ends if you listened to people when they tell you legitimate information... >.>


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 07:49:18 PM
I've been messing around with the shaders in 3DS Max, not the actual files, and although some of it makes it over to Brawlbox, it's easy to say not all of it stays. Even if the Wii is capable of those graphics, Brawl wasn't programmed to have those renders.

I say we focus more on .rel files than shaders! BlackJax has had a lot of progress with the shaders since 0.65 beta, what we really need is more work on the modules and SHP0 like BJ said. Actually BJ, what isn't defined in the SHP0? I thought it had been finished.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 07:52:46 PM
It's possible to do, just not with only a shader.
And maybe you wouldn't run into dead ends if you listened to people when they tell you legitimate information... >.>
yeh...
you do need color values and advanced rendering tecniques in order to pull off what's in the image posted...

believe me when I say, I've actually tried giving Pika fur...
it's anything but simple in terms of simply setting it up D:

you need a vertex group to start off with which will determine the length based on the weighted area...

that's about as far as I've gotten with it before my HDD went RAW...
(I havn't tried it since) <_<

and I'll take that advice, thank you very much XDD


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 07:55:47 PM
I've been messing around with the shaders in 3DS Max, not the actual files, and although some of it makes it over to Brawlbox, it's easy to say not all of it stays. Even if the Wii is capable of those graphics, Brawl wasn't programmed to have those renders.

I say we focus more on .rel files than shaders! BlackJax has had a lot of progress with the shaders since 0.65 beta, what we really need is more work on the modules and SHP0 like BJ said. Actually BJ, what isn't defined in the SHP0? I thought it had been finished.
I really need to know for UMC
(everything must be figured out with no unknowns) D:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: BlackJax96 on January 26, 2012, 07:56:42 PM
I've been messing around with the shaders in 3DS Max, not the actual files, and although some of it makes it over to Brawlbox, it's easy to say not all of it stays. Even if the Wii is capable of those graphics, Brawl wasn't programmed to have those renders.

I say we focus more on .rel files than shaders! BlackJax has had a lot of progress with the shaders since 0.65 beta, what we really need is more work on the modules and SHP0 like BJ said. Actually BJ, what isn't defined in the SHP0? I thought it had been finished.


All that's left is just how all the data structures are linked together through offsets, what the indices do, if there are any flags that need to be figured out to read the data, etc. Then I need to actually parse the data with the notes put together and then rebuild it.

Btw, good luck reverse-engineering rels... They were originally written in C++, which means they're compiled in PPC (I think)... and PPC (PowerPC) is very confusing. We'll really need a team working together on parsing it, along with PW's rel program SRC and the Wii SDK's very small demo of rel parsing (which only really reads the header, unfortunately...) :srs: Somebody will need to read up on PPC too...

ACTUALLY, we could try IDA Pro:
http://en.wikipedia.org/wiki/IDA_Pro

I used it for Xbox 360's XEX game executables when I modded Halo 3 and stuff.


and I'll take that advice, thank you very much XDD


Yay! :happy:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 26, 2012, 08:01:29 PM
Stage modules is a good move, and we do know a bit thanks to PW but I have no idea about the PPC. I could look into that, but it sounds weird to research into a more complex file type before I learn to read file structures from small CHR0 files.

And I dunno about getting the wii to render that fur... isn't Diddy Kong's fur render good enough? Add the bones to it and with Havok's physics engine it'll be fur.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 26, 2012, 08:06:01 PM
sadly...
I'm at the worst position right now to even be looking at brawl data...

once I get my apartment with net on my compy,
I'll be able to help out a little better.

BJ...
I'll be hitting you up for sure ;)
I want to see fur rendered in brawl for my game character "Riku",
and I will make it happen >:3

just like I'll be hacking the WiiU in front of Nintendo's faces if SSB4 comes out as SSBU and is as crappy as everyone says DX

anyways...
I'll try to help as best as I can ;)
but there isn't much I can do w/o the ability to DL/UL


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 27, 2012, 05:20:04 AM
EPIC news on my template :happy:

last night before I went to bed,
I went through and completely redid the BP registers.

the template took about 10 seconds to load FitPikachu00 while out of design mode :happy:

it still takes about an hour to load while in design mode though -.-*

EDIT:
I just did the same thing to the CP registers,
and it now takes 5 seconds to load :happy:

so it now supports everything >:3

it only reads up to the first primitive of each object
(just so you can see the attribute's affect)

speaking of all that,
the facepoints are now read using the attribute values
(the header is ignored)

btw, what's the attribute for '08 00 ########' called??
(I have it labled as 'post')

EDIT2:
so I'm redoing my template base structure now...
(renaming everything)

it'll still support the old method for the resource list offsets...
(until I learn how to do the better method Brbx uses)
^It's more than complicated D:

anyways...
I'm also supporting float values
(now that I know how) 9_9

so yeh...
there's alot to do...

umm...
I won't be adding correct support for Shaders/Nodes until Andrew get's back to me on that...

but everything works perfectly in the area ;)
(it just needs a structure)

if you havn't read in BJ's thread,
the nodes and shaders both follow the format of:
data_size < mult(block_size,32)
(the block size is a multiple of 32 depending on the data size)

well...
unless I can get a valid equation for that here <_<
the template may not support this for a while...
(it takes Andrew a month to reply) D:

EDIT3:
UUUUGH I hate the MDL0 map DX

I knew renaming the var was a bad idea X(
(only as far as verts)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 27, 2012, 11:31:41 AM
Progress for MDL0 data!... although I have this strange feeling that you'll get your MDL0 template up as soon as BlackJax posts his MDL0 structure :P

By the way, that interactive disassembler looks promising. But before I even think about taking on another task, I want Nymph to release his stages for more precious SRT0 unknown data.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DSX8 on January 27, 2012, 12:16:39 PM
Progress for MDL0 data!... although I have this strange feeling that you'll get your MDL0 template up as soon as BlackJax posts his MDL0 structure :P

By the way, that interactive disassembler looks promising. But before I even think about taking on another task, I want Nymph to release his stages for more precious SRT0 unknown data.
LOL  just ask me for some SRT0 files mate :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 27, 2012, 12:51:24 PM
LOL  just ask me for some SRT0 files mate :P
I didn't know if you had all the SRT0 hexed over yet, but if you do send me ALL the SRT0 Data :D Actually, I need the stages so i can test the results from SRT0 edits, so I'll have to wait until you get them done. Send me a download link whenever they're ready!


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 04:58:39 AM
ahah

I just found 1 difference between v10 and v11 MDL0's
(just found my old models of Yami (Okami))

v10 doesn't have v11's tmp3
(as what's supported in my template)

I'll be taking a greater look into these

EDIT:
YES!
I'm so glad I redid the MDL0 map in my template :D

I had so many things with identical names that I was getting an error in the material's
shader that was caused by a mis-interpreted string offset from the resource group D:

at least now I can go through and define a heck of alot more w/o error >:3

ohoho yes, this next release is gonna bomb my last release >:D


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 07:12:17 AM
well...
my template now supports v9 v10 and v11 fully :D
(v10 is basically v11 w/o the unknown3 list value)

EDIT:
I do have a theory that one of the unknown lists would be used for NURBS >_>

I've seen support for NURBS in OBJ files, and even have a few with NURBS and Bezier curve equations...

the curves basically use vertex data for control points,
and have equations that create the inner mesh.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 28, 2012, 08:39:49 AM
wait... there's an entire LIST that's undefined in mdl0? And we didn't need to define it to build mdl0s? How intriguing.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 08:42:06 AM
wait... there's an entire LIST that's undefined in mdl0? And we didn't need to define it to build mdl0s? How intriguing.
look at my new MDL0 thread :D

I have all the lists defined with their corresponding versions

EDIT:
it's actually in my old thread too :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 28, 2012, 09:11:32 AM
Well that's not UNdefined at all, that seems rather defined.

And if so, then you should be ready to move on with module files  :af2:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Eternal Yoshi on January 28, 2012, 09:39:29 AM
Module files require PPC assembly knowledge that only 2 users can even remotely touch and have something good happen.
I'm not one of them.

You may want to start a little slower because that's a very tall order.

I "Somehow" have IDA but don't have a clue on how to use it now.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 28, 2012, 09:40:33 AM
Module files require PPC assembly knowledge that only 2 users can even remotely touch and have something good happen.
I'm not one of them.

You may want to start a little slower because that's a very tall order.


I know, I'm not getting my hopes up. Not even the notes on stage modules will help us too much.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 09:45:34 AM
hey Xiggah...

why don't you help me in figuring out what those undefined lists do :/
(I literally have a u32 TMP data in the place of them)

just see if you can find one where the resource group offset isn't 0 :P

I've got Pokepark to look through yet...
I don't have Mario Kart or Okami =3=

I just want to see if I can get the names changed from unknown#

EDIT:
why don't you wait till I release my template first before starting ;)
(since my template has support for the unknown lists)

unless BJ's brbx supports them <_<

I just need to find an MDL0 that actually uses those lists >_>


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 11:28:18 AM
ok...
so I've put a message in my template:
"if_you_are_seeing_this_contact_me(Tcll5850)_immediatly"

I couldn't use proper punctuation as I put it over a var :P

but you will see it if you have data for the unknown list offsets. ;)
(I'll post an img of what you'll see once I release)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 28, 2012, 12:30:06 PM
Your template, eh? I do a fine job of finding the Brawl files I'm looking for, so I'd be great at helping. I eagerly await your template.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 01:01:16 PM
Your template, eh? I do a fine job of finding the Brawl files I'm looking for, so I'd be great at helping. I eagerly await your template.
yeh...
I'm only 3 TODO's away from an excelent release though ;)
(right now it's only decent)

1 - implement a better method for reading from the resource groups
2 - update the vertex colors to the current method (should look like the vector groups)
3 - implement a mature method for reading from the TEV structures
(currently just reading a specified count for the registers)

but that's all :D

I hated restructuring the MDL0 map the most D:

but yeh...
I can pretty much call this thing done for a decent release :)
now if I can get to a compy with net XD
(wish the wii could UL) >_<


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 28, 2012, 01:05:47 PM
I can't wait, especially if it helps Brawl's mdl0 format a bit. One hunch I have is to search that Trace trophy, that misrendered trophy is just asking for an examination.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 01:14:46 PM
I can&#039;t wait, especially if it helps Brawl&#039;s mdl0 format a bit. One hunch I have is to search that Trace trophy, that misrendered trophy is just asking for an examination.
IKR XD
it's bugging me too :D

I've never seen an MDL0 with that many primitives before >_>


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 03:01:32 PM
just found a small (nothing to worry about) problem with my template...

while reading the registers,
instead of the register being called "BP_MASK", the register will take on the name of the child register...

so you'll end up having:
BP "KSEL[7]"
  BP "KSEL[7]"

the first register is really the mask >_>

again... nothing big...

I have work to do on the section anyways so... blah...


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 05:19:06 PM
well I'm at least doing something IK I can do before releasing my template...

I'm UD-ing the color values to read based on the format...

UUGH!!!
I have to decode RGB565/RGB5A3 data again DDX
I HAAATE that format...
but at least that's the one I got working in my REFT viewer >_>
(it's CMPR that never got finished) DX

anyways...
you should be able to see the actual color values in the structure before the values...

eg:
----Color R,G,B,A = ( 255 , 255 , 255 , 255 )

and also...
unlike the vectors that only parse the first vector,
I'm parsing all the colors as the most I've seen was near 50...
(shouldn't affect the template load time by much at all)

EDIT:
wait...
I got that mixed up...
5+6+5 = 16

I was thinking of RGB5/RGB4A3
(the 16th bit declares the format)


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 06:49:56 PM
I'm currently running tests on different MDL0's and adding extra display data...
(just so you don't have to open the header to see the data)

so far I've had no errors :happy:

and I've changed the "Is_XYZ" data in the vectors back to "has_bounds"

so now that's displayed for Verts, Normals, and UV's.
(all 3 use the same structure)

I've added alot of (but not all) support for float values...
(can't use Design mode to edit my template (have to copy/paste the text))

all that's left on my TODO list is a better method for the Resource Groups, and reading the register data correctly...
(also have to look into the unknown data)

all in all...
this release is about 94% ready =D


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: BlackJax96 on January 28, 2012, 06:52:24 PM
Tcll, for a second there I was trippin' ballz n thought this was one of your threads. :srs:
Maybe you should let someone else post. :P


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 28, 2012, 06:55:26 PM
Tcll, for a second there I was trippin&#039; ballz n thought this was one of your threads. :srs:
Maybe you should let someone else post. :P
lol...
it's been "my" thread for nearly all day :srs:
(nobody's posted)

yeh...
sry for multi-posting though... heh [/bamf]


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 29, 2012, 05:40:22 AM
all night and no reply :srs:

anyways...
in the material header...
has anything been figured out about the P4 and P6 offsets??


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Sky Grounder on January 29, 2012, 06:46:53 AM
all night and no reply :srs:
Well you're mostly talking about MDL0, and MDL0 is so freaking advanced/complicated, so most people don't know much about. That's why they don't reply much.
Thus you always talk about your template :srs:


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 29, 2012, 07:08:04 AM
Well you're mostly talking about MDL0, and MDL0 is so freaking advanced/complicated, so most people don't know much about. That's why they don't reply much.
Thus you always talk about your template :srs:
alright... I'll give ya that then... heh...

and yea...
I'm talking about my template because it's now 96% ready for release :D
(just have a few more things to declare and redefine)

I'll talk more about other stuff when I finish my template ;)

actually... I'll be talking more about UMC development ideas,
but I'll try to keep that more directed to my thread...
(nobody seems to have any ideas to offer though) >_>
^just want to know what to implement to make a decent release is all...

EDIT:
98% ready ;D

just have a little on materials, textures, and pallets to finish


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 29, 2012, 10:50:49 AM
hey Xiggah...
I think I've just looked upon something that's delayed my template...

in the shader (material data) you have the texture clamps and texture matrices...

first off...
I think I have the texture clamps correct in their read order as the data seems to fit...
here's the structure I'm going by:
Angle = f32()
MaxX = f32()
MaxY = f32()
MinX = f32()
MinY = f32()

Min# and Angle are always 0 in the MDL0's I've tested...

anyways...
the texture matrices don't seem to fit the layout >_>

here's what I have:
unk1,unk2,unk3,unk4 = f32(),f32(),f32(),f32()
matrix = mtx_3x4()

I'm getting NAN values that look like they should be:
v1 = s16()
v2 = u16()

note: those values appear randomly throughout the layout...


you can read the brbx src better than me...
so I ask:
what is the proper data layout for both of them??


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 29, 2012, 12:04:54 PM
Well you're mostly talking about MDL0, and MDL0 is so freaking advanced/complicated, so most people don't know much about. That's why they don't reply much.
Thus you always talk about your template :srs:

I know, we could always talk about your event match files. That sounds interesting to look into.


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Sky Grounder on January 29, 2012, 12:06:58 PM
I know, we could always talk about your event match files. That sounds interesting to look into.
I think your reffering to Eternal Yoshi...


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: Xiggah on January 29, 2012, 12:09:33 PM
I think your reffering to Eternal Yoshi...
Right, my bad. SCLO and BLOC data for subspace emissary. Half of the work should be easy if it's just a file container like BRRES right?


Title: Re: - — File Parsing Boot Camp — - (Some file structures added)
Post by: DarkPikachu on January 29, 2012, 12:58:05 PM
99% ready :happy:

and no more TODO data in the shader ;D
(that was the material colors)

all I have to do is add display data to the Textures and Pallets
(not sure how the coords are actually used though) <_<