I have a service that takes a pdf document, resizes all the images, and replaces it in the pdf. The problem that I'm getting at, is the compression.
Some documents are scanned and saved with a Compression.CCITTFAX3 compression and some are saved with a Compression.CCITTFAX4 compression. I am using iTextSharp and convert the stream bytes to a Tiff, otherwise the image becomes funky because of stride or something.
Below is the code I'm currently making use of to check for the correct filter, and then convert to tiff image.
if (filter == "/CCITTFaxDecode")
{
byte[] data = PdfReader.GetStreamBytesRaw((PRStream)stream);
using (MemoryStream ms = new MemoryStream())
{
using (Tiff myTiff = Tiff.ClientOpen("in-memory", "w", ms, new TiffStream()))
{
myTiff.SetField(TiffTag.IMAGEWIDTH, UInt32.Parse(dict.Get(PdfName.WIDTH).ToString()));
myTiff.SetField(TiffTag.IMAGELENGTH, UInt32.Parse(dict.Get(PdfName.HEIGHT).ToString()));
myTiff.SetField(TiffTag.COMPRESSION, Compression.CCITTFAX3);
myTiff.SetField(TiffTag.BITSPERSAMPLE, UInt32.Parse(dict.Get(PdfName.BITSPERCOMPONENT).ToString()));
myTiff.SetField(TiffTag.SAMPLESPERPIXEL, 1);
myTiff.WriteRawStrip(0, data, data.Length);
myTiff.Flush();
using (System.Drawing.Image img = new Bitmap(ms))
{
if (img == null) continue;
ReduceResolution(stream, img, quality);
}
myTiff.Close();
}
}
}
Just to make sure that you understand my question...
I want to find out how I know when to use G3 compression and when to use G4 compression.
Keep in mind that I've tried every code sample I could find.
This is quite important, as we interface with banking systems, and the files uploaded are sent to them as FICA documents.
Please help...
You need to go low level and inspect the image dictionary. The /DecodeParms entry is a dictionary that contains several keys related to CCITT compression. The /K key specifies the compression type: -1 is G4, 0 is G3 1D and 1 is G3 2D.
Update: to be more exact a negative value, usually -1, is G4, 0 is G3 1D and a positive value, usually 1, is G3 2D. To answer your question in the comment, the /K entry is optional and if it is missing the default value is considered to be 0.
I would not advise inserting the data direct. I base this assertion on many years of practical experience of PDFs and TIFF in products like ABCpdf .NET (on which I work).
While in theory you should be able to move the data over direct, minor differences between the formats of the compressed data are likely to lead to occasional mismatches.
The fact that some Fax TIFFs contain data which will display correctly in a TIFF viewer but not in a PDF one leads me to suspect that the same kind of problem is likely to operate in the other direction too.
I'm not going to say this kind of problem is common but it is the kind of thing I wouldn't rely on if I was in a bank. Unless you are very sure your data source will be uniform I would suggest it is much safer to decompress and recompress.
I would also note that sometimes images are held inline in the content stream rather than in a separate XObject. Again this is something you will need to cope with unless your data source produces a standard format which you can be sure will not contain this kind of structure.
Thank you for the replies above. The solution from Mihai seems viable if you do have all the information from the stream. I found that iTextSharp does not do this properly, so I ended up buying pdf4net. Much simpler than trying to figure out whats the better solution, besides, it ended up cheaper than my time I spent on this.
OnceUponATime.... Thank you for the information given above.
PDF4Net has a built in method that you get all the images per page... This sorted my issues, whereas I tried to do this myself using iTextSharp and the examples that were given to me.
Related
Wee bit of background to set the scene : we've told a client he can provide us with images of any type and we'll put them on his reports. I've just had a quick run through of doing this and found that the reports (and other things between me and them) are all designed to only use SVGs.
I thought I'd struck gold when I found online that you can convert an image from a jpg or PNG into an SVG using free tools but alas I've not yet succeeded in getting an SVG stored as bytes in the DB in a format that allows me to use it again after reading it back out.
Here's a quick timeline of what followed leading up to my problem.
I use an online tool to generate an SVG from a PNG (e.g., MobileFish)
I can open and view it in Firefox and it looks ok
I ultimately need the SVG to be stored as bytes in the DB, from where the report will pull it via a webpage that serves it up as SVG. So I write it as bytes into a SQL data script. The code I use to write these bytes is included below.
I visit said webpage and I get an error that there is an "XML parsing error on Line 1 Column 1" and it shows some of my bytes. They begin "3C73"
I revisit the DB and compare the bytes I've just written there with some pre-existing (and working ones). While my new ones begin "3C73", the others begin "0xFFFE".
I think I've probably just pointed out something really fundamental but it hasn't clicked.
Can someone tell me what I've done that means my bytes aren't stored in the correct encoding/format?
When I open my new SVG in Notepad++ I can see the content includes the following which could be relevant :
<image width="900" height="401" xLink:href="data:image/png;base64,
(base 64 encoded data follows for 600+ lines)
Here's the brains of the code that turns my SVG into the bytes to be stored in DB :
var bytes = File.ReadAllBytes(file);
using (var fs = new StreamWriter(file + ".txt"))
{
foreach (var item in bytes)
{
fs.Write(String.Format("{0:X2}",item));
}
}
Any help appreciated.
Cheers
Two things:
SVGs are vector images, not bitmap files. All that online tool is doing is taking a JPEG and creating a SVG file with a JPEG embedded in it. You aren't really getting the benefit of a true SVG image. If you realise and understand that, then no worries.
SVG files are just text. In theory there is no reason you can't just store them as strings in your db. As long as the column is big enough. However normally if you are storing unstructured files in a db, the preferred column type to use is a "Blob".
http://technet.microsoft.com/en-us/library/bb895234.aspx
Converting your SVG file to hex is just making things slower and doubling the size of your files. Also when you convert back, you have to be very careful about the string encoding you are using. Which, in fact, sounds like the problem you are having.
I am suspecting you are doing it incorrectly. SVG is simply and XML based vector image format. I guess your application might be using SVG image element and you need to convert your png image to base64 encoded string .
Not sure if what I'm trying to do will work out, or is even possible. Basically I'm creating a remote desktop type app which captures the screen as a jpeg image and sends it to the client app for displaying.
I want to reduce the amount of data sent each time by comparing the image to the older one and only sending the differences. For example:
var bitmap = new Bitmap(1024, 720);
string oldBase = "";
using (var stream = new MemoryStream())
using (var graphics = Graphics.FromImage(bitmap))
{
graphics.CopyFromScreen(bounds.X, bounds.Y, 0, 0, bounds.Size);
bitmap.Save(stream, ImageFormat.Jpeg);
string newBase = Convert.ToBase64String(stream.ToArray());
// ! Do compare/replace stuff here with newBase and oldBase !
// Store the old image as a base64 string.
oldBase = newBase;
}
Using something like this I could compare both base64 strings and replace any matches. The matched text could be replaced with something like:
[number of characters replaced]
That way, on the client side I know where to replace the old data and add the new. Again, I'm not sure if this would even work so anyones thoughts on this would be very appreciated. :) If it is possible, could you point me in the right direction? Thanks.
You can do this by comparing the bitmap bits directly. Look into Bitmap.LockBits, which will give you a BitmapData pointer from which you can get the pixel data. You can then compare the pixels for each scan line and encode them into whatever format you want to use for transport.
Note that a scan line's length in bytes is always a multiple of 4. So unless you're using 32-bit color, you have to take into account the padding that might be at the end of the scan line. That's what the Stride property is for in the BitmapData structure.
Doing things on a per-scanline basis is easier, but potentially not as efficient (in terms of reducing the amount of data sent) as treating the bitmap as one contiguous block of data. Your transport format should look something like:
<start marker>
// for each scan line
<scan line marker><scan line number>
<pixel position><number of pixels><pixel data>
<pixel position><number of pixels><pixel data>
...
// next scan line
<scan line marker><scan line number>
...
<end marker>
each <pixel position><number of pixels><pixel data> entry is a run of changed pixels. If a scan line has no changed pixels, you can choose not to send it. Or you can just send the scan line marker and number, followed immediately by the next scan line.
Two bytes will be enough for the <pixel position> field and for the <number of pixels> field. So you have an overhead of four bytes for each block. An optimization you might be interested in, after you have the simplest version working, would be to combine blocks of changed/unchanged pixels if there are small runs. For example, if you have uucucuc, where u is an unchanged pixel and c is a changed pixel, you'll probably want to encode the cucuc as one run of five changed pixels. That will reduce the amount of data you have to transmit.
Note that this isn't the best way to do things, but it's simple, effective, and relatively easy to implement.
In any case, once you've encoded things, you can run the data through the built-in GZip compressor (although doing so might not help much) and then push it down the pipe to the client, which would decompress it and interpret the result.
It would be easiest to build this on a single machine, using two windows to verify the results. Once that's working, you can hook up the network transport piece. Debugging the initial cut by having that transport step in the middle could prove very frustrating.
We're currently working on something very similar - basically, what you're trying to implement is video codec (very simple motion jpeg). There are some simple approaches and some very complicated.
The simplest approach is to compare consecutive frames and send only the differences. You may try to compare color differences between the frames in RGB space or YCbCr space and send only the pixels that changed with some metadata.
The more complicated solution is to compare the pictures after DCT transformation but before entropy coding. That would give you better comparisons and remove some ugly artifacts.
Check more info on JPEG, Motion JPEG, H.264 - you may use some methods these codecs are using or simply use the existing codec if possible.
This wont work for a JPEG. You need to use BMP, or possibly uncompressed TIFF.
I think if it were me I'd use BMP, scan the pixels for changes and construct a PNG where everything except the changes were transparent.
First, this would reduce your transmission size because the PNG conpression is quite good especially for repeating pixels.
Second, it makes dispay on the receiving end very easy since you can simply paint the new image overtop the old image.
I'm working on some university project and got stuck with memory issue.
I load a bitmap which takes about 1,5GB on HDD with code below:
Bitmap bmp = new Bitmap(pathToFile);
The issue is that the newly created Bitmap object uses about 3,5GB of RAM which is something I can't understand (that's really BIG wrapper :E). I need to get to the pixel array, and the use of Bitmap class is really helpful (I use LockBits() method later, and process the array byte per byte) but in this case it's total blocker. So here is my question:
Is there any easy way to extract the pixel array without lending additional 2gb?
I'm using c# just to extract the needed array, which is later processed in c++ - maybe I can extract all needed data in c++ (but conversion issue appears here - I'm concentrating on 24bgr format)?
PS: I need to keep the whole bitmap in memory so splitting it into parts is no solution.
PS2: Just to clarify some issues: I know the difference between file extension and file format. The loaded file is uncompressed bitmap 3 bytes per pixel of size ~1.42GB (16k x 32k pixels), so why Bitmap object is more than two times bigger? Any decompressing issues and converting into other format aren't taking place.
Consider using Memory Mapped Files to access your HUGE data :).
An example focused on what you need can be found here: http://visualstudiomagazine.com/articles/2010/06/23/memory-mapped-files.aspx
It's in managed code but you might as well use it from equivalent native code.
Let me know if you need more details.
You can use this solution , Work with bitmaps faster in C#
http://www.codeproject.com/Tips/240428/Work-with-bitmap-faster-with-Csharp
Or you can use memory mapped files
http://visualstudiomagazine.com/articles/2010/06/23/memory-mapped-files.aspx
You can stop memory caching.
Instead of
Bitmap bmp = new Bitmap(pathToFile);
Use
var bmp = (Bitmap)Image.FromStream(sourceFileStream, false, false);
see https://stackoverflow.com/a/47424918/887092
As I know the jpeg file have a best compression ratio between another image extensions, and if I correct we can not more compress a jpeg file because that have best compression, so please help me about this. I create some jpegs as following:
ImageCodecInfo[] codecs = ImageCodecInfo.GetImageEncoders();
ImageCodecInfo ici = null;
foreach(ImageCodecInfo codec in codecs) {
if(codec.MimeType == "image/jpeg")
ici = codec;
}
EncoderParameters ep = new EncoderParameters();
ep.Param[0] = new EncoderParameter(System.Drawing.Imaging.Encoder.Quality, _quality);
using(MemoryStream ms = new MemoryStream()) {
Bitmap capture = GetImage();
capture.Save(ms, ici, ep);
}
And I zipped them with sharpziplib, in average every jpeg size is 130KB and after zip every file compressed to about 70KB, how it possible? there is just 2 answer I can Imagine.
1- We can Compress Jpeg file with more compression ratio by zip libraries
2- My jpeg file not correctly created, and we can create better jpegs (with more compression ratio as we can not more compress them with zip libraries)
Does any one know about this? if we can create better jpegs please help me about it.
Edit:
this is my zip code to compress jpegs:
void addnewentry(MemoryStream stream, string pass,
string ZipFilePath, string entryname){
ICSharpCode.SharpZipLib.Zip.ZipFile zf = new ZipFile(ZipFilePath);
if(!String.IsNullOrEmpty(pass))
zf.Password = pass;
StaticDataSource sds = new StaticDataSource(Stream);
zf.BeginUpdate();
zf.Add(sds, entryName);
zf.CommitUpdate();
zf.IsStreamOwner = true;
zf.Close();
}
public class StaticDataSource : IStaticDataSource {
public Stream stream { get; set; }
public StaticDataSource() {
this.stream.Position = 0;
}
public StaticDataSource(Stream stream) {
this.stream = stream;
this.stream.Position = 0;
}
public Stream GetSource() {
this.stream.Position = 0;
return stream;
}
}
As most of people already stated, you can't compress such already compressed files further easily. Some people works hard on JPEG recompression (recompression = partially decoding already compressed file, and then compressing those data with a custom stronger model and entropy coder. Recompression usually ensures bit-identical results). Even that advanced recompression techniques, I only saw at most 25% improvement. PackJPG is one them. You can have a look at the other compressors here. As you realize, even top rank compressor couldn't reach exactly 25% (even though it's very complex).
Taking these facts into considerations, ZIP (actually deflate) cannot improve compression that much (it's a very old and inefficient if you compare it with top 10 compressors). I believe there are two possible reasons for that problem:
You're accidentally adding some extra data to JPEG stream (possibly adding after JPEG stream).
.NET outputs lots of redundant data to JFIF file. Maybe some big EXIF data and such.
To solve the problem, you can use a JFIF dump tool to observe what's inside the JFIF container. Also, you may want to try your JPEG files with PackJPG.
No one has mentioned that fact that JPEG is merely a container. There are many compression methods that can be used with that file format (JFIF, JPEG-2000, JPEG-LS, etc.) Further compressing that file can yield varying results depending on the content.
Also, some cameras store huge amounts of EXIF data (sometimes resulting in about 20K of data) and that might account for the difference you're seeing.
The JPEG compression algorithm has two stages: a "lossy" stage where visual elements that should be imperceptible to the human eye are removed, and a "lossless" stage where the remaining data is compressed using a technique called Huffmann coding. After Huffmann coding, further lossless compression techniques (like ZIP) will not reduce the size of the image file by significant amount.
However, if you were to zip multiple copies of the same small image together, the ZIP ("DEFLATE") algorithm will recognise the repetition of data, and exploit it to reduce the total file size to less than the sum of the individual files' size. This may be what you're seeing in your experiment.
Stated very simply, losless compression techniques like Huffman coding (part of JPEG) and DEFLATE (used in ZIP) try to discover repeated patterns in your original data, and then represent those repeated patterns using shorter codes.
In short, you won't be able to really improve JPEG by adding on another lossless compression stage.
You can attempt to compress anything with zlib. You just don't always get a reduction in size.
Usually compressing a whole jpeg file will yield a handful of bytes savings as it will compress the jpeg header (including any plain text comments or EXIF data)
This may not fully account for the 40K of compression you see unless you have a huge amount of header data or your jpeg data somehow winds up with alot of repeating values inside.
Zipping JPEGs reduces size because: EXIF data isn't compressed, JPEG is optimized for photos and not GIF-like data, and compressing files creates a single data stream, allowing patterns across multiple files and removing the requirement that each must be aligned with a specific block on disk. The latter can alone save around 4KB per compressed file.
The main problem with zipping pre-compressed images is that it requires extra work (human and CPU) for prep and viewing, which may not be worth the effort (unless you have millions of images that are infrequently accessed, or some kind of automated image service you're developing).
A better approach is to minimize the native file size, forgetting zip. There are many free libraries and apps out there to help with this. For example, ImageOptim combines several libs into one (OptiPNG, PNGCrush, Zopfli, AdvPNG, Gifsicle, PNGOUT), for a barrage of aggressive tricks to minimize size. Works great for PNGs; haven't tried it much with JPEGs.
Though remember that with any compression, there's always a point of diminishing returns. It's up to you to decide whether or not a few extra bytes really matter in the long run.
I'm loading a Bitmap from a jpg file. If the image is not 24bit RGB, I'd like to convert it. The conversion should be fairly fast. The images I'm loading are up to huge (9000*9000 pixel with a compressed size of 40-50MB). How can this be done?
Btw: I don't want to use any external libraries if possible. But if you know of an open source utility class performing the most common imaging tasks, I'd be happy to hear about it. Thanks in advance.
The jpeg should start with 0xFF 0xD8. After that you will find various fields in the format:
Field identifier 2 bytes
Field length, excluding field identifier. 2 bytes.
Variable data.
Parse through the fields. The identifier you will be looking for is 0xFF 0xC0. This is called SOF0, and contains height, width, bit depth, etc. 0xFF 0xC0 will be followed by two bytes for the field length. Immediately following that will be a single byte showing the bit depth, which will usually be 8. Then there will be two bytes for height, two for width, and a single byte for the number of components; this will usually be 1 (for greyscale) or 3. (for color)
This isn't something I've tried myself, but I think you might need to acccess the picture's EXIF information as a start.
Check out Scott Hanselman's blog-entry on accessing EXIF information from pictures.
Standard .NET System.Drawing namespace should have all that you need,
but it probably won't be very efficient. It'll load the whole thing into RAM, uncompress it, convert it (probably by making a copy) and then re-compress and save it. If you aim for high performance, I'm afraid you might need to look into C/C++ libraries and make .NET wrappers for them.
As far as I know jpg is always 24 bpp. The only thing that could change would be that it's CMY(K?) rather then RGB. That information would be stored in the header. Unfortunately I don't have any means of creating a CMYK image to test whether loading into a Bitmap will convert it automatically.
The following line will read the file into memory:
Bitmap image = Image.FromFile(fileName);
image.PixelFormat will tell you the image format. However, I can't test what the file load does with files other than 24bpp RGB jpgs. I can only recommend that you try it out.