Difference between revisions of "Utilities"

From EDukeWiki
Jump to navigation Jump to search
m
 
(46 intermediate revisions by 11 users not shown)
Line 1: Line 1:
When modding for Eduke32 there are some files you will need to edit certain aspects of the game.
When modding EDuke32, there are some files you will need to edit certain aspects of the game.
Here we will cover the needed programs for each aspect.
Here we will cover the needed programs for each aspect.


== Data Files ==


== .GRP Editing ==
=== Group Files (.GRP) ===
.GRP files are basically like a zip file, they hold all of the relevant files for the game, and keep your folders tidy. There are ''numerous'' programs available for use, even some crude extraction and grouping programs came on the Duke3D disc in the goodies directory: "KGROUP" and "KEXTRACT". While lacking a GUI, they get the job done. There are other programs, which can view art files inside the .GRP, and listen to sound bytes. Other programs include, ".GRP File Studio" and Roma Loom's ".GRPViewer".
They hold all of the relevant files for the game, and keep your folders tidy. There are several programs available for use.


==== CLI ====


== .CON Editing ==
The preeminent command-line programs for this purpose, and what the original developers used, are kextract and kgroup, part of the [[Build tool]]s.
.CON files hold most of the important code for the game. To edit the con files, you actually dont need any special programs. Just open the .CON files in your favorite text editor. Special editing programs for writing CON code do exist, though all of them so far have only been developed for use with 1.5 Coding. Therefor, they will cause problems if you try to use Eduke32 exclusive parameters.


We can find a program called grpextract [svn://svn.icculus.org/duke3d/trunk here]. To build it, just do:


== .ART Editing ==
gcc grpextract.c -o grpextract
.ART files hold all the raw art data, in a tidy manner. Once again, there are numerous programs available which can edit the .ART files. The most commonly used program is the one that came with Duke Nukem 3d: Editart. Editart still works on most Windows XP computers. Other programs include, "BastART" and "DukeRES".


While lacking a GUI, they get the job done.


== .MAP Editing ==
==== GUI ====
.MAP files are your levels. There arent too many widely used .MAP editors, the most commonly used and recommended one is Mapster32, which comes with Eduke32. The other commonly used one is Jonof's BUILD, which comes with JFDuke3D. Originally, BUILD from the /Goodies directory on the Duke3D disc was used. Other mapping utilities were made, but are now obsolete, these include "Mapster" and "Makemap".
There are other programs which can view art files inside the .GRP and play sound bytes. Other programs include Jonah Bishop's [https://dukeworld.com/2001-current/rtcm/duke3d-tools-grp/gfs31.zip Group File Studio] and Roma Loom's [[:File:GViewer.zip|GRPViewer]]


For your own mods, it is highly recommended to use either .zip files or folders in conjunction with the -j command line parameter instead of GRP files.


== .PAL Editing ==
=== Zip Files (.zip) ===
.PAL Files hold the palettes, special palettes, and shade tables for the game. This is one of the most loathed and untouched editable features of Duke3D. While incredibly difficult, it can come to good effect. One of the few .PAL editors is the "Build Customization Suite".


For zip files, there are no better than [http://www.7-zip.org/ 7-Zip] (GUI) and [http://advsys.net/ken/utils.htm KZIP] (CLI).


== .ANM Editing ==
For novelty, the .pk3 and .pk4 file extensions are also supported.
.ANM files hold raw .PCX files, which are added together and then played as an animation. In the game these are the animations you see at the end of a boss fight. The sounds for the animations are not included in the .ANM. One of the few .ANM compiling programs is "Animation File Maker".


=== Control Files (.CON) ===
.CON files hold most of the important code for the game. Since they are source code, they are simply text files and can be opened in any text editor. [http://notepad-plus-plus.org/ Notepad++] and [http://www.emeraldeditor.com/ Emerald Editor (aka Crimson Editor)] are recommended.
=== Maps (.MAP) ===
.MAP files are levels. To edit maps, use Mapster32, which comes with EDuke32.
== 8-Bit Resources ==
=== Art Files (.ART) ===
.ART files hold all the raw art data in a tidy manner. Once again, there are numerous programs available which can edit the .ART files. The most commonly used program is the one that came with Duke Nukem 3D: EditArt. It is a 16-bit DOS program, so the use of DOSBox is recommended. Other programs include [https://m210.duke4.net/index.php/downloads/download/8-java/41-build-art-files-editor BAFed (Recommended)], [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/bastART.zip bastART] and [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/dukeresART.zip DukeResART]
The [[Build tool]] '''arttool''' is handy for constructing ART files in a simple and well-defined manner, but be warned that it causes corruption of your art files in some circumstances that has yet to be debugged.
There are two command line utilities called art2tga and tga2art that allow management of art files. NB: They contain some amount of hardcoded behavior to fit the art and palette of the game ''Blood''. [http://hendricks266.duke4.net/stuff/rebuild-bin-win32-101229-h266.7z Win32 binaries], [http://hendricks266.duke4.net/stuff/rebuild-src-101229-h266.7z source code], [http://blood.sourceforge.net/snapshots.php#rebuild-section origin]
There's also a replacement for art2tga/tga2art being written by [[User:Kraigose|Kraigose]] based on their codebase which aims to replace these tools using the more well known PNG format with the simplicity of the TGA tools INI file syntax. A precompiled Win32 build and source can be found [http://www.kraigosestudios.com/programming/eduke32-art-tools/ here] at Kraigose's website. A readme file is included in the download, as well as color tables for Adobe Photoshop. Images must be 256-color PNGs at the moment, but that'll be fixed soon. It's usage syntax is similar to art2tga and tga2art, but it supports by default doing ''all'' ART files in batch. More development is expected.
If you do not wish to finaggle with .art files and their tools, you can use PNG files directly and define them in the DEF language using "tilefromtexture" and related commands. See: [[DEF Language#8-Bit .ART File Replacements]]
=== Animations (.ANM) ===
.ANM files hold raw .PCX files, which are added together and then played as an animation. In the game these are the animations you see at the end of a boss fight. The sounds for the animations are not included in the .ANM. One of the few .ANM compiling programs is [http://dukeworld.duke4.net/rtcm/duke3d-tools-editart/anim2pcx.zip Animation File Maker]
[http://www.ffmpeg.org FFmpeg], since version 0.6,  can also decode and play anm files.
<u>Some examples:</u>
Play the 'foo' animation file:
ffplay foo.anm
Extract all frames from the 'foo' animation file into the PNG format (works also with PCX, and a number of other formats)
ffmpeg -i foo.anm bar%03d.png
It's not possible to encode into ANM format with FFmpeg.
=== Voxels (.KVX) ===
Voxels are simple 3D shapes made out of three-dimensional cubes. While no voxels were ever included in any commercial versions of Duke Nukem 3D, its fellow BUILD games Shadow Warrior and Blood have many. EDuke32 supports voxels as well.
To create and edit voxels, use [http://advsys.net/ken/download.htm Ken Silverman's SLAB6].
=== Palettes (.DAT) ===
The PALETTE.DAT and LOOKUP.DAT files hold the palettes, palettes swaps, shade and translucency (blending) tables for the game. This is one of the most loathed and untouched editable features of Duke3D. While incredibly difficult, it can come to good effect. The [[Build tools]] '''bsuite''', '''mkpalette''', and '''transpal''' can help you in this regard.
=== Sounds (.VOC/.WAV) ===
You are likely going to want to use [[#Audio (.ogg/.flac)|a modern audio format]].
The best tool for the job of the legacy sound formats is [http://www.goldwave.com/ GoldWave]. Save them as 8-bit or 16-bit unsigned PCM, without any form of compression.
=== Music (.MID) ===
As with sounds, you are likely going to want to use [[#Audio (.ogg/.flac)|a modern audio format]].
Composing MIDI files is beyond the scope of this article. However, [http://openmidiproject.sourceforge.jp/Sekaiju_en.html Sekaiju] is useful utility for examining and making objective modifications. [http://musescore.org/ MuseScore] may also be of some use.
A section on MIDI would not be complete without mentioning [http://timidity.sourceforge.net/ TiMidity++].
A very technical description of EMIDI, proprietary extensions to the MIDI format designed for the Apogee Sound System and used in the Duke Nukem 3D MIDI data files, can be found [http://hendricks266.duke4.net/databases/midi/sw/1.2%20Registered%20CD/Emidi11.txt here]. The foo_midi component for foobar2000 can play back EMIDI correctly. Note that EMIDI only works on Windows builds of EDuke32.
== Modern Resources ==
=== Textures (.png/.jpg/.tga/.dds) ===
EDuke32 supports true color textures. These can be made in any image editing program, including Adobe Photoshop, Corel Paint Shop Pro, and [http://www.gimp.org/ The GNU Image Manipulation Program (GIMP)]
Keep in mind that JPEG and DDS are forms of [[Wikipedia:lossy compression|lossy compression]]. Be sure to back up your image in two ways: one, in a PNG-exported final copy, and two, if applicable, in your image editor's source file (PSD/XCF/PSP).
==== PNG Optimization ====
[http://hendricks266.duke4.net/stuff/png_optimize.7z Hendricks266's PNG Optimization Package] contains Windows binaries of the following programs, along with an install script that copies them to system32 and adds right-click entries for optimization scripts to the registry entries of PNG files.
*[http://advsys.net/ken/utils.htm PNGOUT]
*[http://psydk.org/PngOptimizer.php PngOptimizerCL]
=== 3D Models (.md3) ===
==== Programs ====
This is a list of 3D modelling programs which can be used to create and edit 3D models for use with EDuke32.
*[http://www.blender.org Blender], open source 3D modeler can be used for UV unwrapping, texturing, rigging, water simulations, skinning, animating, rendering, particle and other simulations. (Import/Export MD3 scripts for Blender [http://hendricks266.duke4.net/stuff/new_blender_md3.zip 2.43 (up to 2.49)], [http://forums.duke4.net/topic/5358-blender-26-md3-export-script 2.6x+ (by Drek)]. [http://www.katsbits.com/tools See also katsbits.com], they provide several exporters for different versions of Blender.)
*[http://www.misfitcode.com/misfitmodel3d Misfit Model 3D], open source and OpenGL-based 3D model editor that works with triangle-based models (on Debian based distros, package is called mm3d).
*[http://www.turbosquid.com/gmax Gmax], video game modeling tool (Windows, freeware). (MD3 exporter plugin for Gmax can be found [http://www.katsbits.com/tools here].)
*[http://usa.autodesk.com/adsk/servlet/index?siteID=123112&id=5659302 3DS Max], 3D modeler (Windows, commercial). (MD3 export script for 3DS Max [http://www.katsbits.com/tools v4.2/v5] and [http://www.katsbits.com/tools v6].)
*[http://chumbalum.swissquake.ch Milkshape 3D], 3D modeler (Windows, commercial).
==== Tools ====
These are small tools which may prove useful when working with MD3 models.
*[http://www.gamers.org/pub/idgames2/planetquake/q2pmp/NMD3C.zip NPherno's MD3 Compiler]
*[http://automagically.de/g3dviewer/ G3DViewer], open source and OpenGL models viewer, support formats 3D Studio, Quake 2, Quake 3 and many others.
=== Movies (.ivf) ===
IVF files are an alternative to the ANM format utilizing a VP8 video stream, the same used in the WebM format but without the Vorbis audio stream.
To play it in-game, place the ivf file in a place EDuke32 will see and give it the same name as the .anm file that we want to replace (eg ''logo.ivf'' for the animation logo when the game starts).
To encode video, including an uncompressed raw one, to IVF, use this command line syntax with [http://www.ffmpeg.org FFmpeg]:
ffmpeg -i <input> -an -vcodec libvpx -crf 8 -b:v 2M <output>.ivf
See the [https://trac.ffmpeg.org/wiki/Encode/VP8 FFmpeg Encode/VP8 wiki page] for more information about fine-tuning a VP8 encode.
To extract VP8 video tracks from a WebM file into an IVF file, you can use:
ffmpeg -i <input>.webm -an -vcodec copy <output>.ivf
[http://ffmpeg.zeranoe.com/builds/ Windows binaries of FFmpeg] are available.
=== Audio (.ogg/.flac) ===
EDuke32 supports the following modern formats for both sounds and music:
*[http://www.vorbis.com/ Ogg Vorbis]
*[http://flac.sourceforge.net/ FLAC]
Constructing your own sounds and music is beyond the scope of this article, but the following tools may be helpful in processing the audio data into the supported formats:
*Encoders
**Ogg Vorbis
***[http://www.geocities.jp/aoyoume/aotuv/ aoTuv]
***[http://www.rarewares.org/ogg-oggenc.php oggenc2]
**FLAC
***[http://flac.sourceforge.net/documentation_tools.html flac]
*Editors
**[http://www.ffmpeg.org FFmpeg]
**[http://sox.sourceforge.net/ Sound eXchange (SoX)]
**[http://www.goldwave.com/ GoldWave]
**[http://audacity.sourceforge.net/ Audacity]
If you use Vorbis, keep in mind that it is a form of [[Wikipedia:lossy audio compression|lossy audio compression]]. Be sure to back up your source material in FLAC format.
====Looping====
Loops are controlled by three tags in the audio metadata. All values are in '''PCM samples'''.
*'''LOOP_START''' holds the beginning of the loop, the position to which playback returns at the end of the loop.
*'''LOOP_END''' is optional; it holds the end of the loop, after which the game seeks to LOOP_START. If undefined, the end of the file is the end of the loop. The primary purpose of LOOP_END is if you want to give your file a proper ending for listening outside the game.
*'''LOOP_LENGTH''' is another method to define the end of a loop. Internally, LOOP_START + LOOP_LENGTH = LOOP_END. LOOP_END takes precedence over LOOP_LENGTH.
The underscores in all three are optional. All three tags are case-insensitive.
To tag your audio files, use the CLI program [http://www.synthetic-soul.co.uk/tag/ Tag] or the GUI program [http://www.mp3tag.de/en/ Mp3tag].
   
To preview a looped Ogg you have assembled, give it a ".logg" extension and play it using the vgmstream plugin for Winamp or foobar2000. Note that vgmstream only appears to support LOOP_START, not LOOP_LENGTH and LOOP_END. No known program outside of EDuke32 supports looped FLAC.


== Links ==
== Links ==
http://duke3d.bur.st/utilities.php
* [http://dukertcm.com/ R.T.C.M.]
http://www.gamers.org/pub/games/duke3d/other/build/
* [http://dukerepository.com/utilities.php Duke Nukem Repository Utility Page]
* [http://www.gamers.org/pub/games/duke3d/other/build/ Gamers.org Archive Utility Page]
* [http://forums.duke4.net/topic/798-tutorials-resources The Tutorials & Resources thread on the forum]

Latest revision as of 12:04, 5 February 2020

When modding EDuke32, there are some files you will need to edit certain aspects of the game. Here we will cover the needed programs for each aspect.

Data Files

Group Files (.GRP)

They hold all of the relevant files for the game, and keep your folders tidy. There are several programs available for use.

CLI

The preeminent command-line programs for this purpose, and what the original developers used, are kextract and kgroup, part of the Build tools.

We can find a program called grpextract here. To build it, just do:

gcc grpextract.c -o grpextract

While lacking a GUI, they get the job done.

GUI

There are other programs which can view art files inside the .GRP and play sound bytes. Other programs include Jonah Bishop's Group File Studio and Roma Loom's GRPViewer

For your own mods, it is highly recommended to use either .zip files or folders in conjunction with the -j command line parameter instead of GRP files.

Zip Files (.zip)

For zip files, there are no better than 7-Zip (GUI) and KZIP (CLI).

For novelty, the .pk3 and .pk4 file extensions are also supported.

Control Files (.CON)

.CON files hold most of the important code for the game. Since they are source code, they are simply text files and can be opened in any text editor. Notepad++ and Emerald Editor (aka Crimson Editor) are recommended.

Maps (.MAP)

.MAP files are levels. To edit maps, use Mapster32, which comes with EDuke32.

8-Bit Resources

Art Files (.ART)

.ART files hold all the raw art data in a tidy manner. Once again, there are numerous programs available which can edit the .ART files. The most commonly used program is the one that came with Duke Nukem 3D: EditArt. It is a 16-bit DOS program, so the use of DOSBox is recommended. Other programs include BAFed (Recommended), bastART and DukeResART

The Build tool arttool is handy for constructing ART files in a simple and well-defined manner, but be warned that it causes corruption of your art files in some circumstances that has yet to be debugged.

There are two command line utilities called art2tga and tga2art that allow management of art files. NB: They contain some amount of hardcoded behavior to fit the art and palette of the game Blood. Win32 binaries, source code, origin

There's also a replacement for art2tga/tga2art being written by Kraigose based on their codebase which aims to replace these tools using the more well known PNG format with the simplicity of the TGA tools INI file syntax. A precompiled Win32 build and source can be found here at Kraigose's website. A readme file is included in the download, as well as color tables for Adobe Photoshop. Images must be 256-color PNGs at the moment, but that'll be fixed soon. It's usage syntax is similar to art2tga and tga2art, but it supports by default doing all ART files in batch. More development is expected.

If you do not wish to finaggle with .art files and their tools, you can use PNG files directly and define them in the DEF language using "tilefromtexture" and related commands. See: DEF Language#8-Bit .ART File Replacements

Animations (.ANM)

.ANM files hold raw .PCX files, which are added together and then played as an animation. In the game these are the animations you see at the end of a boss fight. The sounds for the animations are not included in the .ANM. One of the few .ANM compiling programs is Animation File Maker

FFmpeg, since version 0.6, can also decode and play anm files.

Some examples:

Play the 'foo' animation file:

ffplay foo.anm

Extract all frames from the 'foo' animation file into the PNG format (works also with PCX, and a number of other formats)

ffmpeg -i foo.anm bar%03d.png

It's not possible to encode into ANM format with FFmpeg.

Voxels (.KVX)

Voxels are simple 3D shapes made out of three-dimensional cubes. While no voxels were ever included in any commercial versions of Duke Nukem 3D, its fellow BUILD games Shadow Warrior and Blood have many. EDuke32 supports voxels as well.

To create and edit voxels, use Ken Silverman's SLAB6.

Palettes (.DAT)

The PALETTE.DAT and LOOKUP.DAT files hold the palettes, palettes swaps, shade and translucency (blending) tables for the game. This is one of the most loathed and untouched editable features of Duke3D. While incredibly difficult, it can come to good effect. The Build tools bsuite, mkpalette, and transpal can help you in this regard.

Sounds (.VOC/.WAV)

You are likely going to want to use a modern audio format.

The best tool for the job of the legacy sound formats is GoldWave. Save them as 8-bit or 16-bit unsigned PCM, without any form of compression.

Music (.MID)

As with sounds, you are likely going to want to use a modern audio format.

Composing MIDI files is beyond the scope of this article. However, Sekaiju is useful utility for examining and making objective modifications. MuseScore may also be of some use.

A section on MIDI would not be complete without mentioning TiMidity++.

A very technical description of EMIDI, proprietary extensions to the MIDI format designed for the Apogee Sound System and used in the Duke Nukem 3D MIDI data files, can be found here. The foo_midi component for foobar2000 can play back EMIDI correctly. Note that EMIDI only works on Windows builds of EDuke32.

Modern Resources

Textures (.png/.jpg/.tga/.dds)

EDuke32 supports true color textures. These can be made in any image editing program, including Adobe Photoshop, Corel Paint Shop Pro, and The GNU Image Manipulation Program (GIMP)

Keep in mind that JPEG and DDS are forms of lossy compression. Be sure to back up your image in two ways: one, in a PNG-exported final copy, and two, if applicable, in your image editor's source file (PSD/XCF/PSP).

PNG Optimization

Hendricks266's PNG Optimization Package contains Windows binaries of the following programs, along with an install script that copies them to system32 and adds right-click entries for optimization scripts to the registry entries of PNG files.

3D Models (.md3)

Programs

This is a list of 3D modelling programs which can be used to create and edit 3D models for use with EDuke32.

  • Blender, open source 3D modeler can be used for UV unwrapping, texturing, rigging, water simulations, skinning, animating, rendering, particle and other simulations. (Import/Export MD3 scripts for Blender 2.43 (up to 2.49), 2.6x+ (by Drek). See also katsbits.com, they provide several exporters for different versions of Blender.)
  • Misfit Model 3D, open source and OpenGL-based 3D model editor that works with triangle-based models (on Debian based distros, package is called mm3d).
  • Gmax, video game modeling tool (Windows, freeware). (MD3 exporter plugin for Gmax can be found here.)
  • 3DS Max, 3D modeler (Windows, commercial). (MD3 export script for 3DS Max v4.2/v5 and v6.)
  • Milkshape 3D, 3D modeler (Windows, commercial).

Tools

These are small tools which may prove useful when working with MD3 models.

Movies (.ivf)

IVF files are an alternative to the ANM format utilizing a VP8 video stream, the same used in the WebM format but without the Vorbis audio stream.

To play it in-game, place the ivf file in a place EDuke32 will see and give it the same name as the .anm file that we want to replace (eg logo.ivf for the animation logo when the game starts).

To encode video, including an uncompressed raw one, to IVF, use this command line syntax with FFmpeg:

ffmpeg -i <input> -an -vcodec libvpx -crf 8 -b:v 2M <output>.ivf

See the FFmpeg Encode/VP8 wiki page for more information about fine-tuning a VP8 encode.

To extract VP8 video tracks from a WebM file into an IVF file, you can use:

ffmpeg -i <input>.webm -an -vcodec copy <output>.ivf

Windows binaries of FFmpeg are available.

Audio (.ogg/.flac)

EDuke32 supports the following modern formats for both sounds and music:

Constructing your own sounds and music is beyond the scope of this article, but the following tools may be helpful in processing the audio data into the supported formats:

If you use Vorbis, keep in mind that it is a form of lossy audio compression. Be sure to back up your source material in FLAC format.

Looping

Loops are controlled by three tags in the audio metadata. All values are in PCM samples.

  • LOOP_START holds the beginning of the loop, the position to which playback returns at the end of the loop.
  • LOOP_END is optional; it holds the end of the loop, after which the game seeks to LOOP_START. If undefined, the end of the file is the end of the loop. The primary purpose of LOOP_END is if you want to give your file a proper ending for listening outside the game.
  • LOOP_LENGTH is another method to define the end of a loop. Internally, LOOP_START + LOOP_LENGTH = LOOP_END. LOOP_END takes precedence over LOOP_LENGTH.

The underscores in all three are optional. All three tags are case-insensitive.

To tag your audio files, use the CLI program Tag or the GUI program Mp3tag.

To preview a looped Ogg you have assembled, give it a ".logg" extension and play it using the vgmstream plugin for Winamp or foobar2000. Note that vgmstream only appears to support LOOP_START, not LOOP_LENGTH and LOOP_END. No known program outside of EDuke32 supports looped FLAC.

Links