Kitty Corp Meow Mix Forums

Adding Cell Shading

So, if you came here, you'll wanting to be making that Wind Waker style cellshade for your import/vertex, right? Well, you came to the right place. If you can do metal, you can most certainly do this.

First off, you'll need this: http://www.mediafire.com/?w7r0dnoo2awa4i9 (http://www.mediafire.com/?w7r0dnoo2awa4i9)

It's a texture, a shader, and a material, all from the Wind Waker Tetra trophy. We're going to be putting this on your character.

ALSO, one last note. Don't do this until your rig is complete. If you re-import, you'll have to do it again.

Part 1: Prep the File

First, import the shadingB texture from the download into your file. It can go right next to all of the rest.

Then, go into your FitCharXX.pac and expand your MDL0. Then open the files Materials and Shaders. If you're working with an official character you're going to be editing the materials all the way up to the ones that end in _ExtMtl. Don't mess with those that end in _ExtMtl though, because those control what the metal box calls. If you're working with an import, you'll simply be editing all of your materials (and there's an awesome tutorial by EternalYoshi and Pikazz for adding metal to your character found here: http://forums.kc-mm.com/index.php?topic=36191.0 (http://forums.kc-mm.com/index.php?topic=36191.0))

NOTE: BB Also has an Auto-Metal adding feature.

Right-click on your MDL0, and click on "Add new shader". If you have an import, it will most likely end up as Shader1. On official .pacs, it might end up as high as (or higher than) shader7.

(http://i844.photobucket.com/albums/ab8/Spex130/BrawlBox%20CellShading%20Tutorial/1.jpg)


Right click on the shader you just made, and click replace. Find the trophy package you downloaded and select "shader0" Now that we've replaced that shader, that's all we'll be doing in the shader folder. You can close it now.

Part 2: Edit the Materials

Now go to your materials folder. Inside, there are a bunch of materials with plusses next to them. Don't expand the materials yet. Instead, click on the first one, and to your right scroll all the way down. Under the MISC bar, you'll find the shader that material is linked to.

(http://i844.photobucket.com/albums/ab8/Spex130/BrawlBox%20CellShading%20Tutorial/2.png)


Change it to the new shader you created. Do that for all the materials that don't end in _ExtMtl. (Or, if you know what you're doing, just do this for all the materials you want to be cellshaded.)

1: Now, go back to the first material, and expand it. Look inside at the sub-material on the first row. On the right, it should tell you what texture that material is using.

(http://i844.photobucket.com/albums/ab8/Spex130/BrawlBox%20CellShading%20Tutorial/3.png)

Take note of this.


2: Now click on the material (not the sub-material) and right-click and replace. Replace this file with "tetra_all" in the trophy material pack.

3: Go to the first sub-material of the material you just replaced, go to the texture reference under misc (as shown in the picture above), and select the texture that was used originally.

4: Then go to the second sub material go to the texture reference box (same spot as above) and type in "shadingB" exactly as shown. (If you've named the shadingB texture anything else, type that in instead.)

Edit(3-11-13): Step 5 is no longer necessary.


Now, that material is done.

For the rest of the materials (all the way up to the ones that end in _ExtMtl, which we aren't going to mess with.), do this:



There. Now your model should be Epic and Cell-Shaded. You might have to edit the included gradient (shadingB) to make it look more cell-shaded, but this is the basic concept.

Lastly, you must remember that you might have to experiment with what materials become cell-shaded. It's not always going to look pretty even if you did it right.

Hopefully it looks nice though.

Here's mine:


One final note:

If an object isn't single-bound the cell-shading will not follow the camera like metal.

It WILL, however, always react to the light correctly. It doesn't matter what way you are facing, the shadows will always be away from the light.[/s]

EDIT: If you imported everything with Forced Floats, it should work fine.

So, that should be it. Ask me questions! Also, this is my first tutorial, so tell me how I did!

I suppose I'll reserve this post. You never know. I've regretted not doing it before, so why not?

This looks pretty straightforward. I wonder what part of the materials are responsible for these effects, but that's easy enough to find out with experimentation.

Haha! The most I can do is tell you how to set it up. How it works is beyond me.

Well, I tried it out with my model. Dunno if I did it wrong, but it works properly (following the light) until the camera moves and then the shader bugs out, depending on which stage I'm on. Probably a loose reference on my part, or maybe it's because I'm in training mode.

"Bugs out" how? It actually might be training mode, but I'm not sure.

Anyways, I'll make sure to post some picture results if I do it right: hopefully I'll get it to follow the camera.

By the way, couldn't this technique have more potential as the ability to use a separate metal texture outside of the metal box? I can imagine a nice Shadow Mario hack using that, or any other character wishing to use a texture that should follow the camera.

I think it should. Honestly, all you'd have to do is this same setup, except you'd be messing with the _ExtMtls instead of the normal materials.

Ok, I think it's time to set this up. Better yet, I might be able to do so in 3DS Max!

Hit me. I'll update the OP as necessary.

It'll have to wait until morning, sorry.

Hehe. Understandable.

thank you for this tutorial. its written pretty well, screens and everything. i wanna try it out on a import sometime.

this is what made yer emerl import 1434934 times kooler.. i really liked it

It looks great, but I hadn't uploaded any pictures yet. Would you like some?

Quote from: Xiggah on January 28, 2012, 08:42:56 AM
It looks great, but I hadn't uploaded any pictures yet. Would you like some?

Wait, what?

I mean the cell-shading works fine

Oh! Right, right!

What was that about the pictures, though?

I was saying I could upload pictures of examples if you wanted some

Oh! Sure! That would be wonderful!

Just tell me where they should go!

Anywhere you want, I suppose. Although I should probably render the shading on another character, my custom models don't have proper normals for showing off this effect...

nice. what about cell-shading with outlines??


Like this:http://myfeldin.com/programming/cellshading.php (http://myfeldin.com/programming/cellshading.php)

Maybe it can be programmed in brawlbox later

I've rendered my models with outlines, it's as simple as duplicating the model, enlarging it, and turning on cullface inside. And working the cell shading works with it too, although I reccomend having a separate shader so the outline isn't cell-shaded as well.

Quote from: Xiggah on February 18, 2012, 09:41:29 AM
I've rendered my models with outlines, it's as simple as duplicating the model, enlarging it, and turning on cullface inside. And working the cell shading works with it too, although I reccomend having a separate shader so the outline isn't cell-shaded as well.

Xiggah's right. It's that simple.

Honestly though, I don't think anything more than Auto-Metal should be added. If EVERYTHING is cell-shaded because it's that easy, the effect loses it's prominence because it doesn't take any work to add it.

Quote from: Xiggah on February 18, 2012, 09:41:29 AM
I've rendered my models with outlines, it's as simple as duplicating the model, enlarging it, and turning on cullface inside. And working the cell shading works with it too, although I reccomend having a separate shader so the outline isn't cell-shaded as well.

im lost here...
i mean that duplicate part. do you mean clone in 3ds max?

Quote from: mr.mugenwii on February 19, 2012, 02:29:30 AM
im lost here...
i mean that duplicate part. do you mean clone in 3ds max?

Yes, we mean in 3DS Max. Although TECHNICALLY speaking, you could be able to export the eye polygons and reimport them as a new polygon and move them in there. But that doesn't feel as well-done as doing it by hand in a modeling program.

On the outline-clone thing, how much bigger should I make the clone? Could I just copy/paste the rig from the original? seeing as its the same thing other than it being bigger?

As for the cell shading, could I use the shaders from an actual Wind Waker model, I kind of want to use that look, since it might be a higher quality shader than the trophy. Or probably the cell shading from CON3 since Lee is actually from that game.

And if possible...could you make a tutorial on making a proper shadow model for imports? You know, for imports that don't come with shadow models? (I mean brawl style shadows)

1)
How big it should be is a matter of preference.  Since it's a clone, you won't really need to edit the Skin Modifier all that much. Just a few tweaks here and there to make sure the outer cell doesn't clip with the inner. (And on that note, it's a bit easier to make an outer Cell AFTER you've rigged it.)

2)
Odds are, you won't be able to use a shader from the ACTUAL Wind Waker game. Diffferent game, different model format. If you really want it to look exact, you can edit the Cell Gradient Texture to make it as you please.

For Rosy the Rascal V2 I edited the Cell Gradient Texture so that it would be more Wind Waker like, as opposed to this Diglett I did which uses the original Cell Gradient Texture and has a bit of a softer side to it.



3)
I've never done that before. I could try after I've tested a bit, but for now go ask Mewtwo2ooo, who made a custom shadow for his Mewtwo 64 import.

well hopefully the double polygon deal with outlines will not mess with the model(size-wise)
im speaking too soon and needs to experiment.lol

I don't see XF Commands, just XF TexGen Flags.

Quote from: Lasimo on March 21, 2012, 12:13:20 AM
I don't see XF Commands, just XF TexGen Flags.

Same thing

Blame BJ, lol.

There was a change in BrawlBox's options since the tutorial was made.

Lol u never mentioned what brawlbox to use
Its brawlbox 65b?

You can use .66b.

I also need to edit the Tutorial. You can skip step 5 with the newest version.

Alright so I tried this but My character was either completely grey or white if it wasn't at a certain angle.

Could it be that I have cull none on or that my model only uses one material?

Quote from: DemonBoy on June 16, 2012, 05:42:13 PM
Could it be that I have cull none on or that my model only uses one material?

Cull shouldn't matter, here.

And what do you mean by only uses one material?

I meant that it only uses one texture for everything.

anyway I got it to work. I just had to edit the shader0 as opposed to adding a new one and replacing that

Hmm. Odd that it would work like that. I'll have to investigate, but congrats and all.

Hey Spex. I followed your tutorial, and updated King DDD's Shaders, but his eyes are messed up. They look like this:

Quote from: Snivy on May 08, 2013, 02:18:28 PM
Hey Spex. I followed your tutorial, and updated King DDD's Shaders, but his eyes are messed up. They look like this:

Sorry but you are not allowed to view spoiler contents.

Looking at Dedede's .Pac file, he uses materials to do eye movements/blinking/looking/etc. If you try to apply Cel-Shading to that, his eyes will break. Completely.

I've also found that trying to set up eye materials like this after you've broken them comes up with some pretty bad results, so to fix this you're just going to have to do everything again, but AVOID replacing Dedede's eye material and shader. That should fix it.

Dedede actually doesn't look half-bad Cel-shaded, haha.

Quote from: Spex130 on May 08, 2013, 02:34:39 PMLooking at Dedede's .Pac file, he uses materials to do eye movements/blinking/looking/etc. If you try to apply Cel-Shading to that, his eyes will break. Completely. I've also found that trying to set up eye materials like this after you've broken them comes up with some pretty bad results, so to fix this you're just going to have to do everything again, but AVOID replacing Dedede's eye material and shader. That should fix it. Dedede actually doesn't look half-bad Cel-shaded, haha.

Oh, that would explain it. {(:{p OK, I'll have to remember that.

Yeah this didn't work at all just made my character grey and messed up his texture. perhaps I should try a different version of BB?  I used .68

Well, I can't really help you if it simply failed. It's never failed for me, haha. What was your character?

sorry if it's a necropost, but the model i'm trying to add a shader to, doesn't actually have the option available D: (it's grayed out)

(btw, i'm trying with Toon Link's bombs)

I looked. Using BB .71, it says the model has some "errors," which is probably why it won't allow you.

If you export and reimport it via DAE, you will be allowed to add shaders and materials, though. It might work. Give it a shot.

Also, on another note, in the later versions of Brawlbox, the shader and material format were changed, so the package provided in the OP won't work with BB .68 on and such.

I'll see what I can do about updating that.

A lot of times when I try to replace materials with "tetra_all" it suddenly says "BrawlBox has stopped responding" for no reason.

Quote from: Spex130 on January 07, 2014, 07:55:39 PM
Also, on another note, in the later versions of Brawlbox, the shader and material format were changed, so the package provided in the OP won't work with BB .68 on and such.

This is probably your problem, since I can't remember if I updated the package or not.

What Brawlbox do you use?

Quote from: Spex130 on January 09, 2012, 09:44:46 PM
Adding Cell Shading

So, if you came here, you'll wanting to be making that Wind Waker style cellshade for your import/vertex, right? Well, you came to the right place. If you can do metal, you can most certainly do this.

First off, you'll need this: http://www.mediafire.com/?w7r0dnoo2awa4i9 (http://www.mediafire.com/?w7r0dnoo2awa4i9)

It's a texture, a shader, and a material, all from the Wind Waker Tetra trophy. We're going to be putting this on your character.

ALSO, one last note. Don't do this until your rig is complete. If you re-import, you'll have to do it again.

Part 1: Prep the File

First, import the shadingB texture from the download into your file. It can go right next to all of the rest.

Then, go into your FitCharXX.pac and expand your MDL0. Then open the files Materials and Shaders. If you're working with an official character you're going to be editing the materials all the way up to the ones that end in _ExtMtl. Don't mess with those that end in _ExtMtl though, because those control what the metal box calls. If you're working with an import, you'll simply be editing all of your materials (and there's an awesome tutorial by EternalYoshi and Pikazz for adding metal to your character found here: http://forums.kc-mm.com/index.php?topic=36191.0 (http://forums.kc-mm.com/index.php?topic=36191.0))

NOTE: BB Also has an Auto-Metal adding feature.

Right-click on your MDL0, and click on "Add new shader". If you have an import, it will most likely end up as Shader1. On official .pacs, it might end up as high as (or higher than) shader7.

(http://i844.photobucket.com/albums/ab8/Spex130/BrawlBox%20CellShading%20Tutorial/1.jpg)


Right click on the shader you just made, and click replace. Find the trophy package you downloaded and select "shader0" Now that we've replaced that shader, that's all we'll be doing in the shader folder. You can close it now.

Part 2: Edit the Materials

Now go to your materials folder. Inside, there are a bunch of materials with plusses next to them. Don't expand the materials yet. Instead, click on the first one, and to your right scroll all the way down. Under the MISC bar, you'll find the shader that material is linked to.

(http://i844.photobucket.com/albums/ab8/Spex130/BrawlBox%20CellShading%20Tutorial/2.png)


Change it to the new shader you created. Do that for all the materials that don't end in _ExtMtl. (Or, if you know what you're doing, just do this for all the materials you want to be cellshaded.)

1: Now, go back to the first material, and expand it. Look inside at the sub-material on the first row. On the right, it should tell you what texture that material is using.

(http://i844.photobucket.com/albums/ab8/Spex130/BrawlBox%20CellShading%20Tutorial/3.png)

Take note of this.


2: Now click on the material (not the sub-material) and right-click and replace. Replace this file with "tetra_all" in the trophy material pack.

3: Go to the first sub-material of the material you just replaced, go to the texture reference under misc (as shown in the picture above), and select the texture that was used originally.

4: Then go to the second sub material go to the texture reference box (same spot as above) and type in "shadingB" exactly as shown. (If you've named the shadingB texture anything else, type that in instead.)

Edit(3-11-13): Step 5 is no longer necessary.

Sorry but you are not allowed to view spoiler contents.

Now, that material is done.

For the rest of the materials (all the way up to the ones that end in _ExtMtl, which we aren't going to mess with.), do this:

Sorry but you are not allowed to view spoiler contents.

There. Now your model should be Epic and Cell-Shaded. You might have to edit the included gradient (shadingB) to make it look more cell-shaded, but this is the basic concept.

Lastly, you must remember that you might have to experiment with what materials become cell-shaded. It's not always going to look pretty even if you did it right.

Hopefully it looks nice though.

Here's mine:

Sorry but you are not allowed to view spoiler contents.

One final note:

If an object isn't single-bound the cell-shading will not follow the camera like metal.

It WILL, however, always react to the light correctly. It doesn't matter what way you are facing, the shadows will always be away from the light.[/s]

EDIT: If you imported everything with Forced Floats, it should work fine.

So, that should be it. Ask me questions! Also, this is my first tutorial, so tell me how I did!

Is there a method for non imported models.

Quote from: Bobomsa64 on October 18, 2016, 01:04:37 AM
Is there a method for non imported models.

Yes. You just do those steps for the model's current shaders.