nsZip - NSP compressor/decompressor to reduce storage

nicoboss · Oct 8, 2019

Halo69 said:
Jaja i went to that link and to be honest i have no idea what i'm looking at, i read the (readme) and i see "requirements" but can't find any instructions on how exactly its done. Unless i'm looking at the wrong thing.

First install python from https://www.python.org/downloads/release/python-374/ and run "python -3 -m pip install -r requirements.txt" inside the main folder to install all the requirements. Then copy your dump and keys.txt into the compressor folder. Inside there execute something like "python nsz.py --level 17 -C dump.nsp"

blawar said:
I'm going to go through my code afterwork, to see if extra data at the very end will break clients. I know it will issue warnings, but it may still be successful. If so, we should move it to the very end of the file, and have a footer rather than a header so it can be read easily and existing clients can use the new format.

I'm digging through the zstandard protocol, it is possible to seek with a single flat stream, but it is indeed a pain in the butt. You are correct that it is probably best from a complexity standpoint to only support seeking with chunked content.

Are you sure you only want to support fixed size chunks? I do not care about seeking, so I will defer to your judgement call on that. I just think variable sized chunks would improve compression ratios at the expense of only a little added complexity. A Binary search would be fast.

With my currently implementation the NCZBLOCKS metadata is just after the NCZSECTN metadata. I have nothing against putting them at the end but kind of useless as the non-solid compressed files won't work with the current implementation of your homebrew anyways. With an updated version of your homebrew we could put them anywhere we want. With my current implementation non-solid compression is an optional future that can be enabled by specifying a block size over command line arguments. If no block size is specified the exact same file will be produced as your tool does it right now.
For the question regarding dynamic decompressed block size. It's would be quite hard to do such a thing on NCA layer as the actual files the game uses are one level deeper. Because of this game files can't be easily recognized as such. Generally, it's quite questionable if it is really worth it. ZStandard is quite good compressing small chunks and not much compression ratio is lost compared to a fully solid compression especially when choosing 512 KB instead of 256 KB as block size. The difference between static chunk and "intelligent" ones would be even lower. For the access time binary search indeed wouldn't be that bad as it's still O(log n) while n will be around 40000 on a 10 GB game resulting in only 14 compression required for it to find the block it needs which is basically nothing compared the I/O and decompression time. With NSZ type 2 I purposed such a non-static dynamic sized format but never implemented it as I just can't see any use of it. Maybe let’s just put an indicator inside NCZBLOCKS for meta version and type in case we ever want to add additional futures it but for now let’s keep the decompressed block size statically.

blawar · Oct 8, 2019

nicoboss said:
First install python from https://www.python.org/downloads/release/python-374/ and run "python -3 -m pip install -r requirements.txt" inside the main folder to install all the requirements. Then compy yur dump and keys.txt into the compressor folder. Inside there execute something like "python nsz.py --level 17 -C tdump.nsp"

With my currently implementation the NCZBLOCKS metadata is just after the NCZSECTN metadata. I have nothing against putting them at the end but kind of useless as the non-solid compressed files won't work with the current implementation of your homebrew anyways. With an updated version of your homebrew we could put them anywhere we want. With my current implementation non-solid compression is an optional future that can be enabled by specifying a block size over command line arguments. If no block size is specified the exact same file will be produced as your tool does it right now.

For the question regarding dynamic decompressed block size. It's would be quite hard to do such a thing on NCA layer as the actual files the game uses are one level deeper. Because of this game files can't be easily recognized as such. Generally, it's quite questionable if it is really worth it. ZStandard is quite good compressing small chunks and not much compression ratio is lost compared to a fully solid compression especially when choosing 512 KB instead of 256 KB as block size. The difference between static chunk and "intelligent" ones would be even lower. For the access time binary search indeed wouldn't be that bad as it's still O(log n) while n will be around 40000 on a 10 GB game resulting in only 14 compression required for it to find the block it needs which is basically nothing compared the I/O and decompression time. With NSZ type 3 I purposed such a non-static dynamic sized format but never implemented it as I just can't see any use of it. Maybe let’s just put an indicator inside NCZBLOCKS for meta version and type in case we ever want to add additional futures it but for now let’s keep the decompressed block size statically.

Hello @nicoboss ,

Thats fine, i'll defer to your judgement, I just wanted to talk it out before we solidified the spec because I loathe changing specs later and breaking backwards compatibility. The spec looks good, with a few minor tweaks:

1) change the magic to 8 bytes (its currently 9) for alignment purposes.
2) change the block size field in the header to the number of bits (8 = 256 bytes, 16 = 65536 bytes etc) to enforce a power of two block size (again for alignment purposes)

I can update the clients this weekend. Unless there are any other changes you want.

Thanks,
-Blake

edit: also 4096 is much too small for a floor on the block size. The switch and the PC have plenty of ram, I would recommend a minimum block size of at least 16MB.

proffk · Oct 10, 2019

Its quite difficult figuring out how to get to blawar script to work. I think a simple gui or tool would make things easier.

Clydefrosch · Oct 10, 2019

so what are the chances any usb installer will support on the fly decompression for installation?

Mthodmn101 · Oct 10, 2019

proffk said:
Its quite difficult figuring out how to get to blawar script to work. I think a simple gui or tool would make things easier.

nsz.py --level 17 -C xxxxx.nsp

thats it

blawar · Oct 10, 2019

Clydefrosch said:
so what are the chances any usb installer will support on the fly decompression for installation?

Tinfoil 5.00 and Nut already support this.

proffk · Oct 10, 2019

Mthodmn101 said:
nsz.py --level 17 -C xxxxx.nsp

thats it

I only asked a logical question. If your going to reply with an attitude like don't bother.

blawar · Oct 10, 2019

proffk said:
I only asked a logical question. If your going to reply with an attitude like don't bother.

I did not read his response as attitude, it looked like he was giving you the example syntax to help you. You should try NSC_Builder, they just added support for NSZ as well.

Mthodmn101 · Oct 10, 2019

proffk said:
I only asked a logical question. If your going to reply with an attitude like don't bother.

I see even reading my simple response was too difficult for you.

--------------------- MERGED ---------------------------

Mthodmn101 said:
I see even reading my simple response was too difficult for you.

Hell, heres a stupid batch script. Rename a text with this to .bat and run

for %%i in (F:\*.nsp) do nsz.py --level 17 -C "%%i"

just change the F:\ to wherever ur nsps are

Halo69 · Oct 12, 2019

nicoboss said:
First install python from https://www.python.org/downloads/release/python-374/ and run "python -3 -m pip install -r requirements.txt" inside the main folder to install all the requirements. Then copy your dump and keys.txt into the compressor folder. Inside there execute something like "python nsz.py --level 17 -C dump.nsp"

With my currently implementation the NCZBLOCKS metadata is just after the NCZSECTN metadata. I have nothing against putting them at the end but kind of useless as the non-solid compressed files won't work with the current implementation of your homebrew anyways. With an updated version of your homebrew we could put them anywhere we want. With my current implementation non-solid compression is an optional future that can be enabled by specifying a block size over command line arguments. If no block size is specified the exact same file will be produced as your tool does it right now.
For the question regarding dynamic decompressed block size. It's would be quite hard to do such a thing on NCA layer as the actual files the game uses are one level deeper. Because of this game files can't be easily recognized as such. Generally, it's quite questionable if it is really worth it. ZStandard is quite good compressing small chunks and not much compression ratio is lost compared to a fully solid compression especially when choosing 512 KB instead of 256 KB as block size. The difference between static chunk and "intelligent" ones would be even lower. For the access time binary search indeed wouldn't be that bad as it's still O(log n) while n will be around 40000 on a 10 GB game resulting in only 14 compression required for it to find the block it needs which is basically nothing compared the I/O and decompression time. With NSZ type 2 I purposed such a non-static dynamic sized format but never implemented it as I just can't see any use of it. Maybe let’s just put an indicator inside NCZBLOCKS for meta version and type in case we ever want to add additional futures it but for now let’s keep the decompressed block size statically.

Ok so i installed python and also downloaded the embeddable zip, but not sure if you mean to run "python -3 -m pip install -r requirements.txt" thru a command promp (which i do but i get an error) or is something i need to find and run (which i cannot find)

Taorn · Oct 12, 2019

Halo69 said:
Ok so i installed python and also downloaded the embeddable zip, but not sure if you mean to run "python -3 -m pip install -r requirements.txt" thru a command promp (which i do but i get an error) or is something i need to find and run (which i cannot find)

Yes, you need to run it from a command line. What did the error say? Did you mark the option "add to path" when installing Python?

Halo69 · Oct 12, 2019

blawar said:
Hello @nicoboss ,

Thats fine, i'll defer to your judgement, I just wanted to talk it out before we solidified the spec because I loathe changing specs later and breaking backwards compatibility. The spec looks good, with a few minor tweaks:

1) change the magic to 8 bytes (its currently 9) for alignment purposes.
2) change the block size field in the header to the number of bits (8 = 256 bytes, 16 = 65536 bytes etc) to enforce a power of two block size (again for alignment purposes)

I can update the clients this weekend. Unless there are any other changes you want.

Thanks,
-Blake

edit: also 4096 is much too small for a floor on the block size. The switch and the PC have plenty of ram, I would recommend a minimum block size of at least 16MB.

@blawar i'm using NSC builder (NSB_093py) after compressing was done, why the NSZ file bigger then the NSP? NSP=606 KB NSZ=764KB but it did shrinked for zelda: NSP= 5.84gb NSZ=2.74gb
So this is basically only to save storage on PC and not on the switch? I installed the NSZ and when i checked the storage on the switch it installed the whole storage of 5.84gb

Taorn said:
Yes, you need to run it from a command line. What did the error say? Did you mark the option "add to path" when installing Python?

Yes i marked that option but when i run that command i get syntax error

Gh0sti · Oct 12, 2019

this is the error I get when doing astral chain did I do something wrong?

Halo69 · Oct 12, 2019

@nicoboss hows that new release coming along where we'll be able to install nspz? Not sure if you going to have one but good to ask. Now im using NSC_builder to compress it to nsz but to be honest that doesnt seem to save to many storage, for example after compression:
NSP: 1.77gb NSZ: 1.61gb
NSP: 559 gb NSZ: 5:45 gb
NSP: 12.9 gb NSZ: 12.6 gb
With those compression sizes might as well just stay with NSP files, all that time it took to compress for barely any save storage. Am i right?

nicoboss · Oct 12, 2019

Gh0sti said:
this is the error I get when doing astral chain did I do something wrong?

The latest pre-release of nsZip is extremely outdated. This error is probably fixed in master so either build master by your own, wait for a new build, ask me for a build and I will send you one or just the NSZ compressor python script from https://github.com/nicoboss/nsz instead and compress into the new NSZ file format. There’s now an option for NSZ to support block based compression and will most likely be the successor to nsZip's currently used NSPZ file format as it's much easier for homebrew developers to support and so is already compatible with blawar Tinfoil 5.00.

Halo69 said:
@nicoboss hows that new release coming along where we'll be able to install nspz? Not sure if you going to have one but good to ask. Now im using NSC_builder to compress it to nsz but to be honest that doesnt seem to save to many storage, for example after compression:
NSP: 1.77gb NSZ: 1.61gb
NSP: 559 gb NSZ: 5:45 gb
NSP: 12.9 gb NSZ: 12.6 gb
With those compression sizes might as well just stay with NSP files, all that time it took to compress for barely any save storage. Am i right?

NSPZ and NSZ booth use the same compression algorithm. If you use NSZ's default sulid compression method at level 18 you should get a better compression ratio then on nsZip at the default compression level "[Lv. 18] Great" if not please report it as in that case something with NSZ is probably wrong. What compression ratio you get highly depends on the game you compress so try some other games and you should see way better results. nsZip will probably switch to the NSZ format in the future so NSZ is probably the way to go.

Halo69 · Oct 12, 2019

nicoboss said:
The latest pre-release of nsZip is extremely outdated. This error is probably fixed in master so either build master by your own, wait for a new build, ask me for a build and I will send you one or just the NSZ compressor python script from https://github.com/nicoboss/nsz instead and compress into the new NSZ file format. There’s now an option for NSZ to support block based compression and will most likely be the successor to nsZip's currently used NSPZ file format as it's much easier for homebrew developers to support and so is already compatible with blawar Tinfoil 5.00.

NSPZ and NSZ booth use the same compression algorithm. If you use NSZ's default sulid compression method at level 18 you should get a better compression ratio then on nsZip at the default compression level "[Lv. 18] Great" if not please report it as in that case something with NSZ is probably wrong. What compression ratio you get highly depends on the game you compress so try some other games and you should see way better results. nsZip will probably switch to the NSZ format in the future so NSZ is probably the way to go.

Oh ok, imma try to switch it to lv 18 cause default is lv 17 and see the results. I kept the threads at 0 cause its the default, should i change that? Also is this only to save space on PC and not on the switch? When i installed a nsz file i saw it installed the actual size.

nicoboss · Oct 12, 2019

Halo69 said:
Oh ok, imma try to switch it to lv 18 cause default is lv 17 and see the results. I kept the threads at 0 cause its the default, should i change that? Also is this only to save space on PC and not on the switch? When i installed a nsz file i saw it installed the actual size.

Lv. 17 vs lv 18 won't make a huge difference but level 17 is dumb anyways as it’s compression speed for compression ratio is pretty bad. Some games just don't compress very well. Threads don't do anything on latest NSZ version. Multithreaded compression for block compression will be implemented in the following days. NSZ currently only saves storage on PC and increases transfer and installation speed. There will probably be NSZ Yuzu support in the future and maybe NSZ CFW integration.

Artyriaboy · Oct 12, 2019

Does this tool use the CPU and GPU for the operations?

nicoboss · Oct 13, 2019

Artyriaboy said:
Does this tool use the CPU and GPU for the operations?

CPU but the current solid compression algorithm is single core. Block compression will be multithreaded soon but will trade off compression ratio for random read access and multithreading. A solid Compression algorithm on GPU wouldn't make any sense as it could only use a single GPU core and so would be way slower than when using a single CPU core - not even sure if implementing zStandard on GPU would even be possible but definitely not something you want to do as GPUs are single instruction multiple data processors while CPUs are multiple instructions multiple data processors. Better use a tenser processing unit which is a multiple instructions single data streaming processor if you really want to use compression specialized hardware – only bad google only lets you rent but not buy them but feel free to implement zStandard on an FPGA board if you really want to. I would interface such specialized hardware in all my software if ever somebody bothers creating a specialized circuit for zStandard but I really don’t think it’s work it as CPUs are pretty good running compression algorithms.

andrewshky · Oct 13, 2019

Is this Windows only? Any plans on making this available for Mac?

Well-Known Member

Developer

Well-Known Member

Well-Known Member

Well-Known Member

Developer

Well-Known Member

Developer

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

iOS Guru

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

New Member

Similar threads

Popular threads in this forum