Debugging HLSL

A lot of guys have asked me for advice on developing and debugging shader programs. Well, it’s a tricky topic to deal with. Tto be able to fully debug shader programs you will need either a shader IDE like FXComposer or a GPU debugging tool like Nvidia Nsight. These are both complex tools and beyond the scope of a quick tutorial but what I can do if provide you a quick guide to aid you in writing shaders directly within Visual Studio. You will not be able to perform any sort of in-depth debugging, but it will help you deal with silly syntax errors. The first thing need is NShader. NShader is a shader syntax highlighting plugin for visual studio and helps with clarity when editing and writing shader programs.

The second thing is to create a custom build step within Visual Studio for your shader programs. This custom build step will use the MS shader compiler to compile your shader programs and notify you of any errors as well as tell you on which lines the errors can be found. To do this, we first select our shader file and right-click then select properties (see figure 1 below).

Figure 1: Shader Code File Properties

Doing so will bring up the properties windows, the first step is to ensure that the configuration drop down is set to “all configurations”. The select Item Type and choose “Custom Build Tool” from the drop down (see figure 2).

Figure 2: Enable Custom Build Tool Step

Click Apply, this will then show the custom build menu tab on the left hand side. Select the tab and you be presented with the following dialog window:

Figure 3: Set Custom Build Tool Parameters

Under the general heading, set the command line value to the following:

"$(DXSDK_DIR)Utilities\bin\x86\"fxc.exe  /T fx_4_0 /Fo "%(Filename).fxo" "%(FullPath)"

This will mean that every time the shader file is modified and the solution is compiled, the shader file will be compiled using the microsoft FX compiler (FXC). The /T flag specifies the type of shader file being compile (i.e. the HLSL version). the /Fo flag refers to the compiled output file and the %(FullPath) macro refers to the full path of the current shader file.

Also set the Custom Build Tool outputs to: $(filename).fxo , this is the same as specified in the FXC commandline. Click OK, and you are done.

The results of the this process is shown below, all HLSL errors will pop up in the Visual Studio Error Dialog, double clicking on the error will take you to the line of code that caused the error.

Figure 4: The results of the Custom Build Step

I’ve wasted a ton of time attempting to find silly syntax errors when developing shader programs, and this little custom build step has been a great help. I hope it helps you in some way as well…


6 thoughts on “Debugging HLSL

  1. Hi Bobby

    I like those kinds of tips, that with little work give a lot of value.

    I have a question – are you sure the parameters the command line for fxc is ok? I’ve copied and pasted it (VS 2008 pro) and I’m getting

    1>Performing Custom Build Step
    1>No files specified, use /? to get usage information
    1>Project : error PRJ0019: A tool returned an error code from “Performing Custom Build Step”

    Also I see there are some differences in the cmdline in post and on screenshot..

  2. the parameters are correct but a lot of people are having trouble cutting and pasting the command line parameters. WordPress has done some sort of character replacement on the quotes. So please replace the “ character with ”

    Update: I’ve edited the blog post so no one else has this problem in the future!

  3. Hmmm, weird, I still have the same error (and I did noticed the problem with quotes and replaced them by myself). Any ideas?

    I’m using VS 2008 pro (I see you have 2010 team system), have the latest (june 2010) DX SDK installed… I’m wondering about the %parametrs – what are they? windows shell variables? there’s no macro like $filename or $fullpath in VS, and so not suprisingly after changing % to $ I got

    1>Project : warning PRJ0018 : The following environment variables were not found:

    OTOH, it’s fishy that in one place there’s $filename and in other %filename…

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s