Hacking Is there some technical info on WBFS scrubbing?

[PsyBlade]

Hi, I would love some info on how exactly WBFS images are scrubbed.

Basically what I need to figure out is a way to determine which blocks of a games are needed and which are not.
I would be very grateful if someone could point me in the right direction.
(Already tried reading sources of wit and wiiscrubber but without a general idea of what was going on neither did the trick)

 

[FAST6191]

Edit- do read the following posts as well.

Basic scrubbing (non fakesigned stuff) is just reading the file location and length table and zeroing out the stuff that is not a file, theoretically Nintendo or a developer could have done something like moving files outside the file table for AP purposes (Sony did it and required DMA type reads instead of iso9660 style reads for some PS1 and PS2 stuff) but it never happened for the Wii.
The basic scrubbing just makes compression easier (the padding is encrypted too which effectively makes it random) and things are still scattered across the image. To that end the more advanced scrubbing (and so the stuff used in the filesystem conversion methods) redoes the iso to have all files without padding (give or take basic stuff like 32 bit or maybe drive sector alignment) one after the other which means you just have a single file with only the game's data and not a lot else which is far smaller.

I do not know the Wii iso format offhand but it is either a location and size affair or location and end of file location affair which which means the maths is not going to be that hard-
In not even pseudocode
Get iso size
Read file allocation table
Calculate file locations and note them
Invert this new map of locations to get blank/padding data and zero them (you now have basic scrubbing)
#more advanced scrubbing
Redo file locations and insert files end to end in new file (maybe want to account for any extra binary information if that ends up changed) and resign.

 

[Cyan]

The first scrubbing method wrote all FF in non used areas.
Then, I think it get changed to all 00 (or is it the other way?)

And then as a third method "safe scrub" was introduced in Wiiscrubber, in prevision of a possible Nintendo AP checking, the beginning of the next sector following a file is not scrubbed.
But I think it was never used by Nintendo to detect a scrubbed image.
I don't know if this method is still in use or if it were reverted to full scrub.

@[member='FAST6191']:
The advanced scrubbing is used to create cISO, right? smaller ISO keeping only used data.

 

[FAST6191]

Ahh yeah I forgot about CISO- traditionally CISO is just a zipped iso (similar logic to CBR and CBZ being rar compressed and zip compressed things for comic book readers) and it would work for scrubbed isos as well; the newly zeroed data would go to next to nothing and the seemingly random encrypted stuff would stay and probably be uncompressed*. What happened in practice I am not sure- I saw some of the early discussions on supporting it in things and then got bored with Wii hacking.

*one of the things discussed was maybe abusing compression flags to ignore the actual data even if it could theoretically be compressed but I doubt that happened in practice.

I also forgot about the "safe scrub" as well, thanks for that Cyan. As for AP the only proper stuff I ever recall seeing (beyond the whole IOS, BCA and system level protection) and an analysis posted of it was one of the iso loaders left a series of bytes in memory which a game searched for and errored out when it detected them (which meant it worked on some others and some older ones), ingenious but truly basic (even the DS did hash checks on the binary).

 

[PsyBlade]

What you call basic is what Im looking for.
I suspected it worked like that but I wasn't sure.

Is some description of the involved formats available?

BTW:
I only want basic scrubbing because that way I can avoid touching most of the game.
Most of the unneded data is already gone as the game is already in wbfs format.
But some wbfs blocks were partially used and padded with zeroes.
Since I now change the block size I can end up with a few blocks that only contain padding and could be dropped.
If I limit it to basic scrubbing I can simply cut them out of the file without ever reading or writing the majority of the game.
Much faster that way.

 

[Wiimm]

Do you know this: http://wit.wiimm.de/info/scrubbing.html#intro

 

[Wiimm]

Here is the meta algorithm:

The Image is divided in blocks, each with 32KiB. A Single layer image have 143432 blocks. LIBWBFS assumes, that there are maximal 2*143432 blocks (enough space for a double layer image).

1. Create a vector of 2*143432 elements (bytes), set all to 0. A 1 represents a used block.
2. Mark the first 10 blocks as used, if at least one byte is not NULL in a block (disc header).
3. Find out all partitions (up to 4 partition tables).
4. For each partition do:
   1. Mark the first 4 blocks as used (TICKET, TMD, CERT, H3).
   2. Remember partition block #4 as BASE.
   3. Find all files of the file system, These are BOOT.BIN, BI.BIN, APPLOADER.IMG, MAIN.DOL, FST.BIN and all files of FST.BIN.
   4. For each file do:
      1. Find partition offset of first and last byte of the file
      2. Divide the offsets by 0x7c00 (rounded down; 0x400 bytes are reserved for checksums) and call them BEGIN and END.
      3. Mark all blocks between BASE+BEGIN and BASE+END (both inclusive) as used.
5. Now we have a list with all used blocks.

 

[PsyBlade]

thanks so far
I hope I'm able to dig out the details (like where and how to read parttable or FST.BIN) out of the sources I have

 

[Wiimm]

// all values are big endian -> use ntohl()

typedef struct wd_fst_item_t
{
union
{
u8  is_dir;
u32 name_off;	// mask with 0x00ffffff
};

u32 offset4;
u32 size;
}
__attribute__ ((packed)) wd_fst_item_t;

fst.bin is a simple list. If the first byte is set to one, it is a directory, otherwise a file. The following 3 bytes are the relative index into the name pool.

The very first item is always a directory. For directories, the 'size' is the "index of the last related file +1". For the very first element, this is equal to the total number of elements.

For the used blocks jobs, ignore the directories, and use the offset4

 

[PsyBlade]

thank you very much
thats exactly the stuff I need

 

[Wiimm]

Be careful with the details. I wrote it from memory without source lookup. But I think, that all is correct.