Utilities: Difference between revisions

From EDukeWiki
Jump to navigation Jump to search
mNo edit summary
 
(11 intermediate revisions by 3 users not shown)
Line 5: Line 5:


=== Group Files (.GRP) ===
=== Group Files (.GRP) ===
.GRP files are basically like a zip file, they hold all of the relevant files for the game, and keep your folders tidy. There are several programs available for use.
They hold all of the relevant files for the game, and keep your folders tidy. There are several programs available for use.
 
==== CLI ====


The preeminent command-line programs for this purpose, and what the original developers used, are kextract and kgroup, part of the [[Build tool]]s.
The preeminent command-line programs for this purpose, and what the original developers used, are kextract and kgroup, part of the [[Build tool]]s.


We can find a program called grpextract [svn://svn.icculus.org/duke3d/trunk here]. To build it, just do:
We can find a program called grpextract [svn://svn.icculus.org/duke3d/trunk here]. To build it, just do:
<code>gcc grpextract.c -o grpextract</code>


While lacking a GUI, they get the job done. There are other programs, which can view art files inside the .GRP, and listen to sound bytes. Other programs include Jonah Bishop's [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-build/gfs31.zip Group File Studio] and Roma Loom's [http://loomsday.com/?page_id=4 GRPViewer]
gcc grpextract.c -o grpextract
 
While lacking a GUI, they get the job done.
 
==== GUI ====
There are other programs which can view art files inside the .GRP and play sound bytes. Other programs include Jonah Bishop's [https://dukeworld.com/2001-current/rtcm/duke3d-tools-grp/gfs31.zip Group File Studio] and Roma Loom's [[:File:GViewer.zip|GRPViewer]]


For your own mods, it is highly recommended to use either .zip files or folders in conjunction with the -j command line parameter instead of GRP files.
For your own mods, it is highly recommended to use either .zip files or folders in conjunction with the -j command line parameter instead of GRP files.
Line 20: Line 26:
For zip files, there are no better than [http://www.7-zip.org/ 7-Zip] (GUI) and [http://advsys.net/ken/utils.htm KZIP] (CLI).
For zip files, there are no better than [http://www.7-zip.org/ 7-Zip] (GUI) and [http://advsys.net/ken/utils.htm KZIP] (CLI).


For kicks, the .pk3 and .pk4 file extensions are also supported.
For novelty, the .pk3 and .pk4 file extensions are also supported.


=== Control Files (.CON) ===
=== Control Files (.CON) ===
Line 31: Line 37:


=== Art Files (.ART) ===
=== Art Files (.ART) ===
.ART files hold all the raw art data in a tidy manner. Once again, there are numerous programs available which can edit the .ART files. The most commonly used program is the one that came with Duke Nukem 3D: EditArt. It is a 16-bit DOS program, so the use of DOSBox is recommended. Other programs include [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/bastART.zip bastART] and [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/dukeresART.zip DukeResART]
.ART files hold all the raw art data in a tidy manner. Once again, there are numerous programs available which can edit the .ART files. The most commonly used program is the one that came with Duke Nukem 3D: EditArt. It is a 16-bit DOS program, so the use of DOSBox is recommended. Other programs include [https://m210.duke4.net/index.php/downloads/download/8-java/41-build-art-files-editor BAFed (Recommended)], [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/bastART.zip bastART] and [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/dukeresART.zip DukeResART]


The [[Build tool]] '''arttool''' is handy for constructing ART files in a simple and well-defined manner, but be warned that it causes corruption of your art files in some circumstances that has yet to be debugged.
The [[Build tool]] '''arttool''' is handy for constructing ART files in a simple and well-defined manner, but be warned that it causes corruption of your art files in some circumstances that has yet to be debugged.
Line 38: Line 44:


There's also a replacement for art2tga/tga2art being written by [[User:Kraigose|Kraigose]] based on their codebase which aims to replace these tools using the more well known PNG format with the simplicity of the TGA tools INI file syntax. A precompiled Win32 build and source can be found [http://www.kraigosestudios.com/programming/eduke32-art-tools/ here] at Kraigose's website. A readme file is included in the download, as well as color tables for Adobe Photoshop. Images must be 256-color PNGs at the moment, but that'll be fixed soon. It's usage syntax is similar to art2tga and tga2art, but it supports by default doing ''all'' ART files in batch. More development is expected.
There's also a replacement for art2tga/tga2art being written by [[User:Kraigose|Kraigose]] based on their codebase which aims to replace these tools using the more well known PNG format with the simplicity of the TGA tools INI file syntax. A precompiled Win32 build and source can be found [http://www.kraigosestudios.com/programming/eduke32-art-tools/ here] at Kraigose's website. A readme file is included in the download, as well as color tables for Adobe Photoshop. Images must be 256-color PNGs at the moment, but that'll be fixed soon. It's usage syntax is similar to art2tga and tga2art, but it supports by default doing ''all'' ART files in batch. More development is expected.
If you do not wish to finaggle with .art files and their tools, you can use PNG files directly and define them in the DEF language using "tilefromtexture" and related commands. See: [[DEF Language#8-Bit .ART File Replacements]]


=== Animations (.ANM) ===
=== Animations (.ANM) ===
.ANM files hold raw .PCX files, which are added together and then played as an animation. In the game these are the animations you see at the end of a boss fight. The sounds for the animations are not included in the .ANM. One of the few .ANM compiling programs is [http://archive.dukertcm.com/knowledge-base/downloads-rtcm/duke3d-tools-editart/anim2pcx.zip Animation File Maker]
.ANM files hold raw .PCX files, which are added together and then played as an animation. In the game these are the animations you see at the end of a boss fight. The sounds for the animations are not included in the .ANM. One of the few .ANM compiling programs is [http://dukeworld.duke4.net/rtcm/duke3d-tools-editart/anim2pcx.zip Animation File Maker]


[http://www.ffmpeg.org FFmpeg], since version 0.6,  can also decode and play anm files.
[http://www.ffmpeg.org FFmpeg], since version 0.6,  can also decode and play anm files.


<u>Some examples :</u>
<u>Some examples:</u>
 
Play the 'foo' animation file :
 
<pre>ffplay foo.anm</pre>
 
Extract all frames from the 'foo' animation file into the PNG format (works also with JPG, PCX, GIF, BMP ...)


<pre>ffmpeg -i foo.anm bar%03d.png</pre>
Play the 'foo' animation file:


Encode the 'foo' animation file into a MPEG-4 format :
ffplay foo.anm


<pre>ffmpeg -i foo.anm bar.avi</pre>
Extract all frames from the 'foo' animation file into the PNG format (works also with PCX, and a number of other formats)
Encode the 'foo' animation file into a WebM format :


<pre>ffmpeg -i foo.anm -threads 2 bar.webm</pre>
ffmpeg -i foo.anm bar%03d.png
 
See 'ffmpeg -codecs' for all available codecs.


It's not possible to encode into ANM format with FFmpeg.
It's not possible to encode into ANM format with FFmpeg.
Line 76: Line 74:
=== Sounds (.VOC/.WAV) ===
=== Sounds (.VOC/.WAV) ===


You are likely going to want to use [[#Audio (.ogg)|OGG Vorbis audio]].
You are likely going to want to use [[#Audio (.ogg/.flac)|a modern audio format]].


The best tool for the job of the legacy sound formats is [http://www.goldwave.com/ GoldWave]. Save them as 8-bit unsigned, without any form of compression.
The best tool for the job of the legacy sound formats is [http://www.goldwave.com/ GoldWave]. Save them as 8-bit or 16-bit unsigned PCM, without any form of compression.


=== Music (.MID) ===
=== Music (.MID) ===


As with sounds, you are likely going to want to use [[#Audio (.ogg)|OGG Vorbis audio]].
As with sounds, you are likely going to want to use [[#Audio (.ogg/.flac)|a modern audio format]].


Composing MIDI files is beyond the scope of this article. However, [http://openmidiproject.sourceforge.jp/Sekaiju_en.html Sekaiju] is useful utility for examining and making objective modifications. [http://musescore.org/ MuseScore] may also be of some use.
Composing MIDI files is beyond the scope of this article. However, [http://openmidiproject.sourceforge.jp/Sekaiju_en.html Sekaiju] is useful utility for examining and making objective modifications. [http://musescore.org/ MuseScore] may also be of some use.
Line 94: Line 92:
=== Textures (.png/.jpg/.tga/.dds) ===
=== Textures (.png/.jpg/.tga/.dds) ===
EDuke32 supports true color textures. These can be made in any image editing program, including Adobe Photoshop, Corel Paint Shop Pro, and [http://www.gimp.org/ The GNU Image Manipulation Program (GIMP)]
EDuke32 supports true color textures. These can be made in any image editing program, including Adobe Photoshop, Corel Paint Shop Pro, and [http://www.gimp.org/ The GNU Image Manipulation Program (GIMP)]
Keep in mind that JPEG and DDS are forms of [[Wikipedia:lossy compression|lossy compression]]. Be sure to back up your image in two ways: one, in a PNG-exported final copy, and two, if applicable, in your image editor's source file (PSD/XCF/PSP).


==== PNG Optimization ====
==== PNG Optimization ====
Line 123: Line 123:
=== Movies (.ivf) ===
=== Movies (.ivf) ===
IVF files are an alternative to the ANM format utilizing a VP8 video stream, the same used in the WebM format but without the Vorbis audio stream.
IVF files are an alternative to the ANM format utilizing a VP8 video stream, the same used in the WebM format but without the Vorbis audio stream.
To play it in-game, place the ivf file in a place EDuke32 will see and give it the same name as the .anm file that we want to replace (eg ''logo.ivf'' for the animation logo when the game starts).


To encode video, including an uncompressed raw one, to IVF, use this command line syntax with [http://www.ffmpeg.org FFmpeg]:
To encode video, including an uncompressed raw one, to IVF, use this command line syntax with [http://www.ffmpeg.org FFmpeg]:


<pre>ffmpeg -i <input> -an -vcodec libvpx -quality best <output>.ivf</pre>
ffmpeg -i <input> -an -vcodec libvpx -crf 8 -b:v 2M <output>.ivf
 
See the [https://trac.ffmpeg.org/wiki/Encode/VP8 FFmpeg Encode/VP8 wiki page] for more information about fine-tuning a VP8 encode.


To extract VP8 video tracks from a WebM file into a IVF file, we can use the program ''mkvextract'', part of MKVToolnix, available [http://www.bunkus.org/videotools/mkvtoolnix here].
To extract VP8 video tracks from a WebM file into an IVF file, you can use:


Command line syntax to use:
ffmpeg -i <input>.webm -an -vcodec copy <output>.ivf
<pre>mkvextract tracks <filename>.webm 1:<filename>.ivf</pre>


To play it in-game now we just have to place it in a search path or in a custom mod directory and give it the same name as the .anm file that we want to replace (eg ''logo.ivf'' for the animation logo when the game starts).
[http://ffmpeg.zeranoe.com/builds/ Windows binaries of FFmpeg] are available.


=== Audio (.ogg) ===
=== Audio (.ogg/.flac) ===


EDuke32 supports OGG Vorbis for both sounds and music.
EDuke32 supports the following modern formats for both sounds and music:


*[http://www.ffmpeg.org FFmpeg]
*[http://www.vorbis.com/ Ogg Vorbis]
*[http://sox.sourceforge.net/ Sound eXchange (SoX)]
*[http://flac.sourceforge.net/ FLAC]
*[http://www.goldwave.com/ GoldWave]
*[http://audacity.sourceforge.net/ Audacity]


As Vorbis is a form of [[Wikipedia:lossy audio compression|lossy audio compression]], be sure to back up your source material in FLAC format.
Constructing your own sounds and music is beyond the scope of this article, but the following tools may be helpful in processing the audio data into the supported formats:
 
*Encoders
**Ogg Vorbis
***[http://www.geocities.jp/aoyoume/aotuv/ aoTuv]
***[http://www.rarewares.org/ogg-oggenc.php oggenc2]
**FLAC
***[http://flac.sourceforge.net/documentation_tools.html flac]
*Editors
**[http://www.ffmpeg.org FFmpeg]
**[http://sox.sourceforge.net/ Sound eXchange (SoX)]
**[http://www.goldwave.com/ GoldWave]
**[http://audacity.sourceforge.net/ Audacity]
 
If you use Vorbis, keep in mind that it is a form of [[Wikipedia:lossy audio compression|lossy audio compression]]. Be sure to back up your source material in FLAC format.


====Looping====
====Looping====


Loops are controlled by three tags in the Vorbis Comment, all in PCM samples.
Loops are controlled by three tags in the audio metadata. All values are in '''PCM samples'''.


*'''LOOP_START''' holds the beginning of the loop, the position to which playback returns at the end of the loop.
*'''LOOP_START''' holds the beginning of the loop, the position to which playback returns at the end of the loop.
*'''LOOP_END''' is optional; it holds the end of the loop, after which the game seeks to LOOP_START. If undefined, the end of the OGG is the end of the loop. The primary purpose of LOOP_END is if you want to give your file a proper ending for listening outside the game.
*'''LOOP_END''' is optional; it holds the end of the loop, after which the game seeks to LOOP_START. If undefined, the end of the file is the end of the loop. The primary purpose of LOOP_END is if you want to give your file a proper ending for listening outside the game.
*'''LOOP_LENGTH''' is another method to define the end of a loop. Internally, LOOP_START + LOOP_LENGTH = LOOP_END. LOOP_END takes precedence over LOOP_LENGTH.
*'''LOOP_LENGTH''' is another method to define the end of a loop. Internally, LOOP_START + LOOP_LENGTH = LOOP_END. LOOP_END takes precedence over LOOP_LENGTH.


The underscores in all three are optional. All Vorbis Comment tags are case-insensitive.
The underscores in all three are optional. All three tags are case-insensitive.
 
To tag your audio files, use the CLI program [http://www.synthetic-soul.co.uk/tag/ Tag] or the GUI program [http://www.mp3tag.de/en/ Mp3tag].
      
      
To preview a looped OGG you have assembled, give it a ".logg" extension and play it using the vgmstream plugin for Winamp or foobar2000. Note that vgmstream only appears to support LOOP_START, not LOOP_LENGTH and LOOP_END.
To preview a looped Ogg you have assembled, give it a ".logg" extension and play it using the vgmstream plugin for Winamp or foobar2000. Note that vgmstream only appears to support LOOP_START, not LOOP_LENGTH and LOOP_END. No known program outside of EDuke32 supports looped FLAC.
 
To tag your OGG files, use the CLI program [http://www.synthetic-soul.co.uk/tag/ Tag] or the GUI program [http://www.mp3tag.de/en/ Mp3tag].


== Links ==
== Links ==

Latest revision as of 13:04, 5 February 2020

When modding EDuke32, there are some files you will need to edit certain aspects of the game. Here we will cover the needed programs for each aspect.

Data Files

Group Files (.GRP)

They hold all of the relevant files for the game, and keep your folders tidy. There are several programs available for use.

CLI

The preeminent command-line programs for this purpose, and what the original developers used, are kextract and kgroup, part of the Build tools.

We can find a program called grpextract here. To build it, just do:

gcc grpextract.c -o grpextract

While lacking a GUI, they get the job done.

GUI

There are other programs which can view art files inside the .GRP and play sound bytes. Other programs include Jonah Bishop's Group File Studio and Roma Loom's GRPViewer

For your own mods, it is highly recommended to use either .zip files or folders in conjunction with the -j command line parameter instead of GRP files.

Zip Files (.zip)

For zip files, there are no better than 7-Zip (GUI) and KZIP (CLI).

For novelty, the .pk3 and .pk4 file extensions are also supported.

Control Files (.CON)

.CON files hold most of the important code for the game. Since they are source code, they are simply text files and can be opened in any text editor. Notepad++ and Emerald Editor (aka Crimson Editor) are recommended.

Maps (.MAP)

.MAP files are levels. To edit maps, use Mapster32, which comes with EDuke32.

8-Bit Resources

Art Files (.ART)

.ART files hold all the raw art data in a tidy manner. Once again, there are numerous programs available which can edit the .ART files. The most commonly used program is the one that came with Duke Nukem 3D: EditArt. It is a 16-bit DOS program, so the use of DOSBox is recommended. Other programs include BAFed (Recommended), bastART and DukeResART

The Build tool arttool is handy for constructing ART files in a simple and well-defined manner, but be warned that it causes corruption of your art files in some circumstances that has yet to be debugged.

There are two command line utilities called art2tga and tga2art that allow management of art files. NB: They contain some amount of hardcoded behavior to fit the art and palette of the game Blood. Win32 binaries, source code, origin

There's also a replacement for art2tga/tga2art being written by Kraigose based on their codebase which aims to replace these tools using the more well known PNG format with the simplicity of the TGA tools INI file syntax. A precompiled Win32 build and source can be found here at Kraigose's website. A readme file is included in the download, as well as color tables for Adobe Photoshop. Images must be 256-color PNGs at the moment, but that'll be fixed soon. It's usage syntax is similar to art2tga and tga2art, but it supports by default doing all ART files in batch. More development is expected.

If you do not wish to finaggle with .art files and their tools, you can use PNG files directly and define them in the DEF language using "tilefromtexture" and related commands. See: DEF Language#8-Bit .ART File Replacements

Animations (.ANM)

.ANM files hold raw .PCX files, which are added together and then played as an animation. In the game these are the animations you see at the end of a boss fight. The sounds for the animations are not included in the .ANM. One of the few .ANM compiling programs is Animation File Maker

FFmpeg, since version 0.6, can also decode and play anm files.

Some examples:

Play the 'foo' animation file:

ffplay foo.anm

Extract all frames from the 'foo' animation file into the PNG format (works also with PCX, and a number of other formats)

ffmpeg -i foo.anm bar%03d.png

It's not possible to encode into ANM format with FFmpeg.

Voxels (.KVX)

Voxels are simple 3D shapes made out of three-dimensional cubes. While no voxels were ever included in any commercial versions of Duke Nukem 3D, its fellow BUILD games Shadow Warrior and Blood have many. EDuke32 supports voxels as well.

To create and edit voxels, use Ken Silverman's SLAB6.

Palettes (.DAT)

The PALETTE.DAT and LOOKUP.DAT files hold the palettes, palettes swaps, shade and translucency (blending) tables for the game. This is one of the most loathed and untouched editable features of Duke3D. While incredibly difficult, it can come to good effect. The Build tools bsuite, mkpalette, and transpal can help you in this regard.

Sounds (.VOC/.WAV)

You are likely going to want to use a modern audio format.

The best tool for the job of the legacy sound formats is GoldWave. Save them as 8-bit or 16-bit unsigned PCM, without any form of compression.

Music (.MID)

As with sounds, you are likely going to want to use a modern audio format.

Composing MIDI files is beyond the scope of this article. However, Sekaiju is useful utility for examining and making objective modifications. MuseScore may also be of some use.

A section on MIDI would not be complete without mentioning TiMidity++.

A very technical description of EMIDI, proprietary extensions to the MIDI format designed for the Apogee Sound System and used in the Duke Nukem 3D MIDI data files, can be found here. The foo_midi component for foobar2000 can play back EMIDI correctly. Note that EMIDI only works on Windows builds of EDuke32.

Modern Resources

Textures (.png/.jpg/.tga/.dds)

EDuke32 supports true color textures. These can be made in any image editing program, including Adobe Photoshop, Corel Paint Shop Pro, and The GNU Image Manipulation Program (GIMP)

Keep in mind that JPEG and DDS are forms of lossy compression. Be sure to back up your image in two ways: one, in a PNG-exported final copy, and two, if applicable, in your image editor's source file (PSD/XCF/PSP).

PNG Optimization

Hendricks266's PNG Optimization Package contains Windows binaries of the following programs, along with an install script that copies them to system32 and adds right-click entries for optimization scripts to the registry entries of PNG files.

3D Models (.md3)

Programs

This is a list of 3D modelling programs which can be used to create and edit 3D models for use with EDuke32.

  • Blender, open source 3D modeler can be used for UV unwrapping, texturing, rigging, water simulations, skinning, animating, rendering, particle and other simulations. (Import/Export MD3 scripts for Blender 2.43 (up to 2.49), 2.6x+ (by Drek). See also katsbits.com, they provide several exporters for different versions of Blender.)
  • Misfit Model 3D, open source and OpenGL-based 3D model editor that works with triangle-based models (on Debian based distros, package is called mm3d).
  • Gmax, video game modeling tool (Windows, freeware). (MD3 exporter plugin for Gmax can be found here.)
  • 3DS Max, 3D modeler (Windows, commercial). (MD3 export script for 3DS Max v4.2/v5 and v6.)
  • Milkshape 3D, 3D modeler (Windows, commercial).

Tools

These are small tools which may prove useful when working with MD3 models.

Movies (.ivf)

IVF files are an alternative to the ANM format utilizing a VP8 video stream, the same used in the WebM format but without the Vorbis audio stream.

To play it in-game, place the ivf file in a place EDuke32 will see and give it the same name as the .anm file that we want to replace (eg logo.ivf for the animation logo when the game starts).

To encode video, including an uncompressed raw one, to IVF, use this command line syntax with FFmpeg:

ffmpeg -i <input> -an -vcodec libvpx -crf 8 -b:v 2M <output>.ivf

See the FFmpeg Encode/VP8 wiki page for more information about fine-tuning a VP8 encode.

To extract VP8 video tracks from a WebM file into an IVF file, you can use:

ffmpeg -i <input>.webm -an -vcodec copy <output>.ivf

Windows binaries of FFmpeg are available.

Audio (.ogg/.flac)

EDuke32 supports the following modern formats for both sounds and music:

Constructing your own sounds and music is beyond the scope of this article, but the following tools may be helpful in processing the audio data into the supported formats:

If you use Vorbis, keep in mind that it is a form of lossy audio compression. Be sure to back up your source material in FLAC format.

Looping

Loops are controlled by three tags in the audio metadata. All values are in PCM samples.

  • LOOP_START holds the beginning of the loop, the position to which playback returns at the end of the loop.
  • LOOP_END is optional; it holds the end of the loop, after which the game seeks to LOOP_START. If undefined, the end of the file is the end of the loop. The primary purpose of LOOP_END is if you want to give your file a proper ending for listening outside the game.
  • LOOP_LENGTH is another method to define the end of a loop. Internally, LOOP_START + LOOP_LENGTH = LOOP_END. LOOP_END takes precedence over LOOP_LENGTH.

The underscores in all three are optional. All three tags are case-insensitive.

To tag your audio files, use the CLI program Tag or the GUI program Mp3tag.

To preview a looped Ogg you have assembled, give it a ".logg" extension and play it using the vgmstream plugin for Winamp or foobar2000. Note that vgmstream only appears to support LOOP_START, not LOOP_LENGTH and LOOP_END. No known program outside of EDuke32 supports looped FLAC.

Links