TPL - file specification
Nov 13, 2021 15:52:57 GMT 10
Re-Play games, Biohazard4X, and 4 more like this
Post by zatarita on Nov 13, 2021 15:52:57 GMT 10
Hello! I'm new here. I'm typically a Halo modder, but recently a few people in your community reached out to me for some assistance with a custom tool. While working on the tool I learned a lot about the TPL format, and I figured I would share it with you. HardRain made a tutorial here about it, and I want to add to it.
There are a few unique things to note about TPL files. A few files in the game are actually big endian. These DATs contain TPLs that are also big endian. These big endian TPLs are a different version from the ones located through out the rest of the game. These TPLs are NOT covered here.
The TPL file is a image format similar to TGA. A TPL file can contain multiple child TPLs inside of it. Each child TPL has it's own header which dictates how the pixel data is stored in stream. There are three distinct modes in which the TPL functions. Two of them are similar, and one is unique; Each has it's own perks. Mode 8 and Mode 9 are both palette based formats. Indices are stored for each pixel in the bitmap. These indices point to a color in the palette (or Color LookUp Table (CLUT) ). Mode 6 is a True-Color mode which doesn't use a color palette. Mode 6 is a sequential array of pixel data where the indices would be. A TPL can contain up to 2 mipmaps. Each mipmap has it's own TPL header as well; however, the color palette used for the mipmap shares the original images color palette. Because of this there is no palette in the mipmaps' header.
Unfortunately there are still some unknowns; however, the information is complete enough to be able to extract images from TPLs.
With the high level over view out of the way, let's take a more technical look:
Lets Get Technical:
The TPL will be broken into 3 parts:
TPL Header (red) -> Entry Header(blue) -> Entry Data(green)
TPL Header:
Since the TPL is a container. It can hold one texture, or many textures. the TPL header tells us how many Entries there are in the TPL. In this example here I picked a simple TPL with only one texture inside of it.
The TPL Header is always 0x10 bytes wide. If we break down the header we have three parts the header guard(green), entry count(blue), and offset to first(red).
Header Guard: This is an assumption; however, I believe the 0x1000 is there to act as a header guard to validate the file was read correctly from stream. If not it is still always 0x1000 so it doesn't matter.
Entry Count: How many entries there are in the file. This DOES NOT include mipmaps, as mipmaps are children of entries, not entries themselves.
Offset To First: This will always be 0x10 as the header is always this wide.
The only value that changes in 99% of TPLs is the entry count. The other 1% is the big endian version of the TPL not covered here.
Entry Header:
The Entry Header comes after the TPL Header. There is one per entry count defined in the TPL Header. Each header is 0x30 bytes wide. This determines the width (yellow), height (green), mipmap count(purple), mode(cyan), interlacing(blue) indices offset(orange), palette offset(red), and other metadata about the texture itself.
The width and the height, as well as the mode determine how many indices there are in the stream. By carefully balancing you can maximize space efficiency while maintaining texture quality.
Mode 6 uses no palette, and is a stream of 32-bit color pixels.
Mode 8 uses 4-bit indices to point to a palette of 16 colors.
Mode 9 uses 8-bit indices to point to a palette of 256 colors.
This means the space efficiency of each image can be calculated for width/height and will be constant regardless.
Mode 6: 4 * (Width * Height)
Mode 8: ((Width * Height) \ 2) + (16 * 4) + 0x40(padding)
Mode 9: (Width * Height) + (256 * 4)
Also worth pointing out is interlacing is supported; however, my knowledge on interlacing is limited.
0 = bgra
1 = bgra + 1
2 = PS2 Swizzle.
Mode 6 textures also don't have a palette offset as seen here:
For every dat file in the game that is little endian, I have not seen any other options besides the ones listed above. I created a program to go through and record values that have been set throughout every dat, these are the only ones I've seen.
note that the mipmap count can't always be trusted as some mode 6 textures have mipmaps without having the count as 2. I typically determine if a texture has mipmaps by checking to see if the offsets are non-zero. Speaking of mipmaps...
Textures With Mipmaps
Textures can also contain mipmaps, mipmaps are smaller versions of the texture used to save resources for textures far away. remember that these mipmaps are children of the main textures, and don't count towards to total entry count.
In this example we see in blue the mipmap count is 2, we have two highlights in red, and green. These two highlighted values are offsets that point to the mipmaps Entry Header. If we follow it we see another header:
Notice how the palette offset is 0 for this entry. This is because the mipmap uses the offset pointed to by the parent. in this case 0x19090 seen below the green highlight.
Sadly everything else in the Entry Header is either unknown, or left out due to uncertainty.
Entry Data:
The Entry Data consists of two parts, the indices, and the palette. To start I'm going to look at the palette.
Each (non-mipmap) texture has a palette offset, if we follow the offset we are brought to the end of the indices to a list of BGRA pixels. BGRA is just RGBA with the R and B swapped. Depending on mode there will be a different number of pixels
4-bit: 16 colors with 0x20 bytes of padding in-between (I accidently swapped red/green, magenta = alpha)
8-bit: 256 colors with no padding (I accidently swapped red/green, magenta = alpha)
The palette is loaded into an array, and given an index for it's position. Regardless of the TPL mode, the color is always 32-bit.
The alpha channel is also signed for some reason. So 100% opacity is 0x80 not 0xff
Indices:
The indices are a list of which color to pick from the palette
4-bit indices:
4 bit indices are half of a byte. They are also swapped for some reason. In this case the first pixel is 2, then the second is 1. So instead of reading red, blue. Read it blue, red. These numbers point to a color in the palette. There are exactly width * height indices in a file; One index per pixel. This means for the first pixel in the image, use the second color in the palette. For the second pixel use the first color, then use the second color, then the second color again...so on. Due to the fact that each index is 4 bits, this means the highest indexable number is 0x0f. Which is why the palette only has 16 colors.
8-bit indices:
8 bit indices use the entire byte. They are streamed in order though, so it reads blue, red, blue, red. The same thing happens here as in the 4-bit palette, except the highest indexable palette color is 0xff which is why the palette holds 256 colors.
True Color: (I accidently swapped red/green, magenta = alpha)
True color textures don't use indices. Instead it has and array of 32-bit bgra colors. These colors are similar to the palette colors except the alpha isn't signed. This is practically a TGA file with a custom header, and the blue and red channels swapped.
If we apply these ideas we can extract textures from the game:
There are a few unique things to note about TPL files. A few files in the game are actually big endian. These DATs contain TPLs that are also big endian. These big endian TPLs are a different version from the ones located through out the rest of the game. These TPLs are NOT covered here.
The TPL file is a image format similar to TGA. A TPL file can contain multiple child TPLs inside of it. Each child TPL has it's own header which dictates how the pixel data is stored in stream. There are three distinct modes in which the TPL functions. Two of them are similar, and one is unique; Each has it's own perks. Mode 8 and Mode 9 are both palette based formats. Indices are stored for each pixel in the bitmap. These indices point to a color in the palette (or Color LookUp Table (CLUT) ). Mode 6 is a True-Color mode which doesn't use a color palette. Mode 6 is a sequential array of pixel data where the indices would be. A TPL can contain up to 2 mipmaps. Each mipmap has it's own TPL header as well; however, the color palette used for the mipmap shares the original images color palette. Because of this there is no palette in the mipmaps' header.
Unfortunately there are still some unknowns; however, the information is complete enough to be able to extract images from TPLs.
With the high level over view out of the way, let's take a more technical look:
Lets Get Technical:
The TPL will be broken into 3 parts:
TPL Header (red) -> Entry Header(blue) -> Entry Data(green)
TPL Header:
Since the TPL is a container. It can hold one texture, or many textures. the TPL header tells us how many Entries there are in the TPL. In this example here I picked a simple TPL with only one texture inside of it.
The TPL Header is always 0x10 bytes wide. If we break down the header we have three parts the header guard(green), entry count(blue), and offset to first(red).
Header Guard: This is an assumption; however, I believe the 0x1000 is there to act as a header guard to validate the file was read correctly from stream. If not it is still always 0x1000 so it doesn't matter.
Entry Count: How many entries there are in the file. This DOES NOT include mipmaps, as mipmaps are children of entries, not entries themselves.
Offset To First: This will always be 0x10 as the header is always this wide.
The only value that changes in 99% of TPLs is the entry count. The other 1% is the big endian version of the TPL not covered here.
Entry Header:
The Entry Header comes after the TPL Header. There is one per entry count defined in the TPL Header. Each header is 0x30 bytes wide. This determines the width (yellow), height (green), mipmap count(purple), mode(cyan), interlacing(blue) indices offset(orange), palette offset(red), and other metadata about the texture itself.
The width and the height, as well as the mode determine how many indices there are in the stream. By carefully balancing you can maximize space efficiency while maintaining texture quality.
Mode 6 uses no palette, and is a stream of 32-bit color pixels.
Mode 8 uses 4-bit indices to point to a palette of 16 colors.
Mode 9 uses 8-bit indices to point to a palette of 256 colors.
This means the space efficiency of each image can be calculated for width/height and will be constant regardless.
Mode 6: 4 * (Width * Height)
Mode 8: ((Width * Height) \ 2) + (16 * 4) + 0x40(padding)
Mode 9: (Width * Height) + (256 * 4)
Also worth pointing out is interlacing is supported; however, my knowledge on interlacing is limited.
0 = bgra
1 = bgra + 1
2 = PS2 Swizzle.
Mode 6 textures also don't have a palette offset as seen here:
For every dat file in the game that is little endian, I have not seen any other options besides the ones listed above. I created a program to go through and record values that have been set throughout every dat, these are the only ones I've seen.
note that the mipmap count can't always be trusted as some mode 6 textures have mipmaps without having the count as 2. I typically determine if a texture has mipmaps by checking to see if the offsets are non-zero. Speaking of mipmaps...
Textures With Mipmaps
Textures can also contain mipmaps, mipmaps are smaller versions of the texture used to save resources for textures far away. remember that these mipmaps are children of the main textures, and don't count towards to total entry count.
In this example we see in blue the mipmap count is 2, we have two highlights in red, and green. These two highlighted values are offsets that point to the mipmaps Entry Header. If we follow it we see another header:
Notice how the palette offset is 0 for this entry. This is because the mipmap uses the offset pointed to by the parent. in this case 0x19090 seen below the green highlight.
Sadly everything else in the Entry Header is either unknown, or left out due to uncertainty.
Entry Data:
The Entry Data consists of two parts, the indices, and the palette. To start I'm going to look at the palette.
Each (non-mipmap) texture has a palette offset, if we follow the offset we are brought to the end of the indices to a list of BGRA pixels. BGRA is just RGBA with the R and B swapped. Depending on mode there will be a different number of pixels
4-bit: 16 colors with 0x20 bytes of padding in-between (I accidently swapped red/green, magenta = alpha)
8-bit: 256 colors with no padding (I accidently swapped red/green, magenta = alpha)
The palette is loaded into an array, and given an index for it's position. Regardless of the TPL mode, the color is always 32-bit.
The alpha channel is also signed for some reason. So 100% opacity is 0x80 not 0xff
Indices:
The indices are a list of which color to pick from the palette
4-bit indices:
4 bit indices are half of a byte. They are also swapped for some reason. In this case the first pixel is 2, then the second is 1. So instead of reading red, blue. Read it blue, red. These numbers point to a color in the palette. There are exactly width * height indices in a file; One index per pixel. This means for the first pixel in the image, use the second color in the palette. For the second pixel use the first color, then use the second color, then the second color again...so on. Due to the fact that each index is 4 bits, this means the highest indexable number is 0x0f. Which is why the palette only has 16 colors.
8-bit indices:
8 bit indices use the entire byte. They are streamed in order though, so it reads blue, red, blue, red. The same thing happens here as in the 4-bit palette, except the highest indexable palette color is 0xff which is why the palette holds 256 colors.
True Color: (I accidently swapped red/green, magenta = alpha)
True color textures don't use indices. Instead it has and array of 32-bit bgra colors. These colors are similar to the palette colors except the alpha isn't signed. This is practically a TGA file with a custom header, and the blue and red channels swapped.
If we apply these ideas we can extract textures from the game:
