Develop Reference

imageresizer.net performance benchmark

imageresing.net community and developers.
Please, clarify me some details about imageresing.net internals.
Does imageresing.net use .NET Drawing library to recompress jpegs? If not - does it use a 3rd party engine or some internal algorithms?
Are there performance benchmarks? I'd like to compare imageresing.net with other libraries: libjpeg, Intel Integrated Performance Primitives, etc.
Thanks in advance,
Anton

ImageResizer offers 3 imaging pipelines.
GDI+ (System.Drawing) The default. 2-pass high-quality bicubic with pre-smoothing. Very fast for the quality provided. (Avg. 200-300ms for resizing.) A recent windows update makes GDI+ parallelize poorly, but this is being actively investigated by MSFT.
WIC (What WPF also uses). 4-8x faster (20-200ms for resizing). Single-pass resizing causes lots of moire artifacts and causes blurriness (in both Fant and Bicubic modes). No really high quality resizing is available within Windows Imaging Components.
FreeImage. If you need to support DSLR formats or access Lanczos resampling (top-tier quality), FreeImage is your library. It's slower than the others (often 800-2400ms), large, monolithic, and hard to audit, so we only recommend using it with trusted image data. We use a custom version of FreeImage built against libjpeg-turbo, which is significantly faster than libjpeg.
You can mix and match encoders, decoders, and (to some extent) resizing algorithms. Some algorithms are implemented internally for quality reasons, while most are implemented in C/C++ in dependencies.
End-to-end comparison benchmarking is kind of absurd if you care about photo quality, since you can never compare apples to apples. Back in 2011, I did some benchmarks between GDI+ and WIC, but photographers and graphics designers tend to find WIC image quality unacceptable, so it's not particularly fair.
We regularly benchmark each pipeline against itself to detect performance improvements or regressions, but comparing pipelines can be deceptive for a numer of reasons:
Do you care about metadata? libjpeg-turbo is (wierdly) 2-3x faster at reading jpegs when you disable metadata parsing. If you want to auto-rotate based on camera exif data, you'll need that info.
Do you care about color correctness? Jpeg isn't an RGB format. If it has an ICC profile, the right thing to do is convert to sRGB before you resize. That's slow.
Do you care about resizing quality? There are a hundred ways to implement a bicubic resizing filter. Some fast, some slow, most ugly, some accurate. Bicubic WIC != Bicubic GDI+. You can get < 20ms end-to-end out of ImageResizer WIC in nearest neighbor mode - if you're cool with the visual results.
Do you care about output file size? If you're willing to spend more clock cycles, you can gain 30-80% reductions in PNG/GIF file sizes, and 5-15% in jpeg. If you want to add 150-600ms per request, ImageResizer can create WebP images that halve your bandwidth costs (WebP is more costly to encode than jpeg).
You can make sense of micro-benchmarks (is libjpeg-turbo 40% faster than libjpeg under the same circumstances, etc). You can even compare certain simple low-quality image resizing filters (nearest-neighbor, box, bilinear) after excluding encoding, decoding, and color transformation.
The problem is that truly high-quality resizing is really complex, and never implemented the same way twice. There are a vanishingly small number of high-quality implementations, and an even tinier number that have sub-second performance. I ordered a dozen textbooks on image processing to see if I could find a reference implementation, but the topic is... expertly avoided by most, and only briefly touched on by others. Edge-pixel handling, pre-filtering, and performance optimization are never mentioned.
I've funded a lot of research into fast high-quality image resizing, but we haven't been able to match GDI+ yet. ImageResizer's default configuration tends to beat Photoshop quality on many types of images.
A fourth pipeline may be added to ImageResizer in the near future, based on our fork of libgd with custom resizing algorithms. No promises yet, but we may have something nearly as high quality as GDI+ with similar single-threaded (but better concurrent) performance.
All our source code is on GitHub, so if you find something fast that you'd like to demo as a plugin or alternate pipeline, we'd love to hear about it.

Fastest PNG decoder for .NET

Our web server needs to process many compositions of large images together before sending the results to web clients. This process is performance critical because the server can receive several thousands of requests per hour.
Right now our solution loads PNG files (around 1MB each) from the HD and sends them to the video card so the composition is done on the GPU. We first tried loading our images using the PNG decoder exposed by the XNA API. We saw the performance was not too good.
To understand if the problem was loading from the HD or the decoding of the PNG, we modified that by loading the file in a memory stream, and then sending that memory stream to the .NET PNG decoder. The difference of performance using XNA or using System.Windows.Media.Imaging.PngBitmapDecoder class is not significant. We roughly get the same levels of performance.
Our benchmarks show the following performance results:
Load images from disk: 37.76ms 1%
Decode PNGs: 2816.97ms 77%
Load images on Video Hardware: 196.67ms 5%
Composition: 87.80ms 2%
Get composition result from Video Hardware: 166.21ms 5%
Encode to PNG: 318.13ms 9%
Store to disk: 3.96ms 0%
Clean up: 53.00ms 1%
Total: 3680.50ms 100%
From these results we see that the slowest parts are when decoding the PNG.
So we are wondering if there wouldn't be a PNG decoder we could use that would allow us to reduce the PNG decoding time. We also considered keeping the images uncompressed on the hard disk, but then each image would be 10MB in size instead of 1MB and since there are several tens of thousands of these images stored on the hard disk, it is not possible to store them all without compression.
EDIT: More useful information:
The benchmark simulates loading 20 PNG images and compositing them together. This will roughly correspond to the kind of requests we will get in the production environment.
Each image used in the composition is 1600x1600 in size.
The solution will involve as many as 10 load balanced servers like the one we are discussing here. So extra software development effort could be worth the savings on the hardware costs.
Caching the decoded source images is something we are considering, but each composition will most likely be done with completely different source images, so cache misses will be high and performance gain, low.
The benchmarks were done with a crappy video card, so we can expect the PNG decoding to be even more of a performance bottleneck using a decent video card.

There is another option. And that is, you write your own GPU-based PNG decoder. You could use OpenCL to perform this operation fairly efficiently (and perform your composition using OpenGL which can share resources with OpenCL). It is also possible to interleave transfer and decoding for maximum throughput. If this is a route you can/want to pursue I can provide more information.
Here are some resources related to GPU-based DEFLATE (and INFLATE).
Accelerating Lossless compression with GPUs
gpu-block-compression using CUDA on Google code.
Floating point data-compression at 75 Gb/s on a GPU - note that this doesn't use INFLATE/DEFLATE but a novel parallel compression/decompression scheme that is more GPU-friendly.
Hope this helps!

Have you tried the following 2 things.
1)
Multi thread it, there is several ways of doing this but one would be a "all in" method. Basicly fully spawn X amount of threads, for the full proccess.
2)
Perhaps consider having XX thread do all the CPU work, and then feed it to the GPU thread.
Your question is very well formulated for being a new user, but some information about the senario might be usefull?
Are we talking about a batch job or service pictures in real time?
Do the 10k pictures change?
Hardware resources
You should also take into account what hardware resources you have at your dispoal.
Normaly the 2 cheapest things are CPU power and diskspace, so if you only have 10k pictures that rarly change, then converting them all into a format that quicker to handle might be the way to go.
Multi thread trivia
Another thing to consider when doing multithreading, is that its normaly smart to make the threads in BellowNormal priority.So you dont make the entire system "lag". You have to experiment a bit with the amount of threads to use, if your luck you can get close to 100% gain in speed pr CORE but this depends alot on the hardware and the code your running.
I normaly use Environment.ProcessorCount to get the current CPU count and work from there :)

I've written a pure C# PNG coder/decoder ( PngCs ) , you might want to give it a look.
But I higly doubt it will have better speed permance [*], it's not highly optimized, it rather tries to minimize the memory usage for dealing with huge images (it encodes/decodes sequentially, line by line). But perhaps it serves you as boilerplate to plug in some better compression/decompression implementantion. As I see it, the speed bottleneck is zlib (inflater/deflater), which (contrarily to Java) is not implemented natively in C# -I used a SharpZipLib library, with pure C# managed code; this cannnot be very efficient.
I'm a little surprised, however, that in your tests decoding was so much slower than encoding. That seems strange to me, because, in most compression algorithms (perhaps in all; and surely in zlib) encoding is much more computer intensive than decoding.
Are you sure about that?
(For example, this speedtest which read and writes 5000x5000 RGB8 images (not very compressible, about 20MB on disk) gives me about 4.5 secs for writing and 1.5 secs for reading). Perhaps there are other factor apart from pure PNG decoding?
[*] Update: new versions (since 1.1.14) that have several optimizations; if you can use .Net 4.5, specially, it should provide better decoding speed.

You have mutliple options
Improve the performance of the decoding process
You could implement another faster png decoder
(libpng is a standard library which might be faster)
You could switch to another picture format that uses simpler/faster decodeable compression
Parallelize
Use the .NET parallel processing capabilities for decoding concurrently. Decoding is likely singlethreaded so this could help if you run on multicore machines
Store the files uncompressed but on a device that compresses
For instance a compressed folder or even a sandforce ssd.
This will still compress but differently and burden other software with the decompression. I am not sure this will really help and would only try this as a last resort.

What is the difference between zlib's gzip compression and the compression used by .NET's GZipStream?

Having an odd problem - one of my app suites has to read/write gzip-compressed files that are used on both Windows and Linux, and I am finding that the files I generate using zlib on Linux are 2-3 times larger than those I generate using GZipStream on Windows. They read perfectly on either platform, so I know that the compression is correct regardless of which platform created the file. The thing is, the files are transferred across the network at various times, and obviously file size is a concern.
My question is:
Has anyone else encountered this
Is there some documented difference between the two? I do know that GZipStream does not provide a way to specify the compression level like you can with zlib, but I am using maximum compression on the zlib side. Shouldn't I see relatively the same file size, assuming that GZipStream is written to use maximum compression as well?

And the answer is .... the Linux version was never compressing the data to begin with. Took a lot of debugging to find the bug that caused it, but after correcting it, the sizes are now comparable on both platforms.

I think the reason you are experiencing this is not because of the compression algorithm used, but because of how the files are compressed. From the zLib manual:
"The zlib format was designed to be compact and fast for use in memory and on communications channels. The gzip format was designed for single- file compression on file systems, has a larger header than zlib to maintain directory information, and uses a different, slower check method than zlib."
I think what is happening is that the files on your linux machine are being Tar'red together into 1 file, then that one file is being compressed. In WIndows, I think it compresses each individual file, then stores them compressed into 1 file.
This is my theory, but have nothing to really support it. Thought I might try some trial tests at home later, just to fill my curiosity.

C#: Seeking PNG Compression algorithm/library [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
I need to compress or at least drop the quality of some png images that users are uploading to my site. I already resized it but that doesn't do much for the image size.
Seeking a png/image compression or quality loss algorithm or library for .net 4.0 or under.
This is how I currently save/convert my images:
Image mainImg = ImageHelper.ResizeImage(bmp, 600, 500, false);
mainImg.Save(filePath, System.Drawing.Imaging.ImageFormat.Png);

Unfortunately, .NET (and GDI+, which the .NET graphics libraries are built on) does not support any encoding parameters for PNG. In fact, if you call GetEncoderParameterList on your image with the PNG encoder Clsid, you'll get a "Not implemented" exception.
Even more unfortunate is that both ImageFormat and ImageCodecInfo are sealed classes and you can't just add new codecs to what .NET can natively access. Microsoft dropped the ball on this one.
This means your alternatives are, in decreasing order of masochism:
1) Implement a save function on your own in .NET that implements RFC 2083,
2) Implement a save function based off porting libpng to .NET,
3) Call unmanaged code that was built from libpng directly
libpng has great documentation and is free, both in availability and license permissiveness.

PNG is generally a lossless compression scheme, which means image quality is never reduced unless you reduce the pixel count (size) or color depth. There are three ways to reduce the file size of PNG images:
Reduce the pixel count (by downscaling the image); there's an obvious loss of resolution here
Using a different precompression filters/DEFLATE compressor parameters (OptiPNG is my favorite tool to automatically choose the best combination of filters/compressor settings)
Reducing the color depth from true color to 256 (or less) colors will give you substantial byte savings, but usually at great visual cost (especially for photographic images)
It seems like .NET doesn't have a built-in way to change precompression filters or compressor parameters of PNGs, so you'll have to use an external library. OptiPNG or any of the other PNG optimization tools aren't native .NET libraries, so you'll have to deal with P/Invoking the libraries or running a separate process to.. process each image. However, you're often looking at a savings around 5% or less (although I've seen as much as 40%) since this is ultimately a lossless process.
I'd recommend against reducing color depth in the automated scenario you describe, because the results can look truly awful (think dithered GIFs of old). (However, If you're doing manual optimization, look at Color Quantizer on Windows and ImageAlpha on Mac.)
Realistically, the best way you can reduce file size at the expense of quality is to just convert to JPEG, which requires no external dependencies, and is designed to be a lossy compression algorithm:
mainImg.Save(filePath, System.Drawing.Imaging.ImageFormat.Jpeg); // make sure you change `filePath` to end with jpg instead of png.

Unfortunately, .Net's image processor for png won't do any optimization heuristics on the output. Your best bet is to have an optipng/pngcrush binary available on the server, render the resized output to a temporary file, then use pngcrush on it via a System.Diagnostics.Process.
For the most part, if the uploads are photographs and the original format is JPEG, then you will be better served by using JPEG output.

Create zip style file without compression

I know that there are many free and not so free compression libraries out there, but for the project i am working on, i need to be able to take file data from a stream and put it into some kind zip or pack file, but without compression, because i will need to access these files quickly without having to wait for them to decompress.
Anyone know how this could be approached, or if there are some libraries out there that do this that i am not aware of?

You can use Zip for this. You would use a compression level of something like "none" or "store", which just combines the files without compression. This site enumerates some of them:
Maximum - The slowest of the
compression options, but the most
useful for creating small archives.
Normal - The default value.
Low - Faster than the default, but
less effective.
Minimum - Extremely fast
compression, but not as efficient as
other methods.
None - Creates a ZIP file but does
not compress it. File size may be
slightly larger if archive is
encrypted or made self-extracting.
Here are some C# examples:
CodeProject
EggheadCafe
For the unix unaware, this is exactly what tar does. When you see .tar.gz files, it's just a bunch of files combined into a tar file, and then run through gzip.

Have a look at System.IO.Packaging namespace.
Quote from MSDN:
System.IO.Packaging
Provides classes that support storage
of multiple data objects in a single
container.
Package is an abstract class that can
be used to organize objects into a
single entity of a defined physical
format for portability and efficient
access.
A ZIP file is the primary physical
format for the Package. Other Package
implementations might use other
physical formats such as an XML
document, a database, or Web service.
You can select different compression options for your package:
NotCompressed - Compression is turned off.
Normal - Compression is optimized for a balance between size and
performance.
Maximum - Compression is optimized for
size.
Fast - Compression is optimized for
performance.
SuperFast - Compression is optimized for high performance.

Perhaps just use a zip with compression set to "none"; SharpZipLib would suffice.
Be careful about assuming that compression is slower, though - it might actually (depending on the scenario) be quicker with compression, since you reduce the amount of physical IO and IPC (often a bottleneck), and simply do a bit more CPU work; but you generally have plenty of CPU.

Traditionally simple storage files under windows are cabinet files, which do support compression, as well as signing, what zip does not support.
Look out if theres a way to create cabinet files within .net.

Remember to profile first. Your harddrive is much slower than your cpu or ram. If the file is sitting on disk reading a smaller, compressed file will take less time than if you read an uncompressed blob. The difference may well be more than the time it takes to decompress it.
Also the OS may cache the file in memory. When that happens the harddrive is completely removed from the loop (transparent to you). That could make the decompression time too costly.
I learned this "technique" when dealing with slow internet connections. The client needed the data fast and we had cycles to spare. Sending compressed packets increase the throughput/latency of the application.

I had an additional requirement that the resulting pack file be browsable with standard tools (at least FAR Manager).
So far I've tried:
OPC (Open Packaging Conventions, System.Packaging namespace, ZIP-based, backend for MSO .docx files). Built-in and standard, but quite slow, probably because it actually first copies all data to a temporary location in case it has to be compressed (even if it's not so), and only then writes to the final destination. Unbearably slow. Note that there's also a Windows built-in implementation which is not .NET-based, might be faster but does not span all of the OS versions I have to support.
ITSS (InfoTech Storage System, the backend of CHM files). Built-in in Windows, somewhat standard. Surprisingly, the implementation is incomplete, and it's deadly slow, even slower than OPC.
DOC (COM Compound File Structured Storage, backend for MSO .doc files, .msi files, etc). Built-in in Windows, quite standard. Does not support file names longer than 32 characters, which is a significant drawback in my case. Fast enough on small to middle sizes (totally outruns .NET OPC impl), but has some scalability issues when it goes up to gigabytes.
Various ZIP implementations are still to be tested.