Saturday, January 9, 2010

Content Pipeline

The content/data pipeline in my engine is pretty simple and fast. Two main data folders exists, one for raw data (intermediate) and one for pre-processed data (build). Raw data would be .tga, .png, .fbx and so on and pre-processed data is in a engine specific format.

The tool for pre-processing data from intermediate to build is called jdpp (jmb data pre-processor). jdpp can be executed on specific file or directory. Typically when you sync data you have to run a batch file which executes jdpp and updates the build folder.

When the editor is running a background thread is constantly monitoring the intermediate folder for changes and any change will automatically be reflected in the build folder. For instance, any resource file (model, texture) that is saved it will automatically reloaded in the editor.

In my first attempt each resource type had it's own pre-processor exe but it turned out to be horribly slow to execute an exe 10 000 times. I gained a lot of speed simply by having one master pre-processor exe (jdpp).

Yesterday I also added support for utilizing all CPU cores when processing data. I even gained about 30-40% speed on my single core hyper-threaded netbook. I guess mainly because there are a lot of I/O blocking going on...

No comments:

Post a Comment