This is my first question, so I hope to provide what it needs to get a decent answer.
I want to send an image received by a webcam over a serial link.
The Image is converted into a byte array and then written to the serial port.
The first issue I ran into was, that when I tried to send the image, it lead to a TimeoutException. Looking at the lenght of the byte array, it showed me around 1 MB of data that needs to be transmitted. Shrinking the actual size of the image resulted in an much faster transmission, but afterwards the image was way too small.
The second isuue was when I tried to compress the image. Using different methods, the size of transmission was always excactly the same.
I hope you can help me find a way to improve my implementation, so that the transmission only takes a few seconds while still maintaining reasonable resolution of the image. Thanks.
Specific Information
Webcam Image
The image from the webcam is received by the AForge library
The image is handled as a Bitmap
(Obviously) it doesn't transmit every frame, only on the click of a button
Serial Port
The port uses a baud rate of 57600 bps (defined by hardware beneath)
The WriteTimeout-value is set to 30s, as it would be unacceptable to wait longer than that
Text transmission works with default values on the SerialPort-item in a WinForm
Image Manipulation
I used different approaches to compress the image:
Simple method like
public static byte[] getBytes(Bitmap img)
{
MemoryStream ms = new MemoryStream();
img.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
byte[] output = ms.toArray();
ms.Dispose();
return output;
}
as well as more advanced methods like the one posted here. Not only with Encoder.Quality but also with Encoder.Compression.
My Application
private void btn_Send(...)
{
Bitmap currentFrame = getImageFromWebcam();
//Bitmap sendFrame = new Bitmap(currentFrame, new Size(currentFrame.Width/10, currentFrame.Height/10));
Bitmap sendFrame = compressImage(currentFrame);
byte[] data = getBytes(sendFrame);
serialPort.Write(data, 0, data.Lenght);
}
C Hanging the timeout property of the serial port would solve the timeout issue. How is show in this link https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.writetimeout(v=vs.110).aspx. File compression works by looking at blocks of data and associating similar blocks with each other for a given segment of blocks. If your image is too unique it will not compress depending on the compression software being used.
Related
I need to take a Pillow image and either convert it to a byte array or pack it somehow to send it over a ZeroMQ socket. Since the other socket connection isn't in python (it's c#), I can't use pickle, and I'm not sure JSON would work since it's just the image (dimensions sent separately). I'm working with an image created from a processed numpy array out of opencv, so I don't have an image file. Sending the numpy array over didn't work, either.
Since I can't read the image bytes from a file, I've tried Image.getdata() and Image.tobytes() but I'm not sure either were in the right form to send over the socket. What I really need is a bytestream that I can reform into an array after crossing the socket.
UPDATE: So now I'm specifically looking for anything easily undone in C#. I've looked into struct.pack but I'm not sure there's an equivalent to unpack it. Turning the image into an encoded string would work as long as I could decode it.
UPDATE 2: I'm thinking that I could use JSON to send a multidimensional array, then grab the JSON on the other side and turn it into a bitmap to display. I'm using clrzmq for the C# side, though, and I'm not sure how that handles JSON objects.
In case anyone was wondering, here's what I ended up doing:
On the python side:
_, buffer = cv2.imencode('.jpg', cFrame )
jpg_enc = base64.b64encode(buffer).decode('utf-8')
self.frameSender.send_string(jpg_enc)
self.frameSender.send_string(str(height))
self.frameSender.send_string(str(width))
On the C# side:
byte[] bytebuffer = Convert.FromBase64String(frameListener.ReceiveFrame().ReadString());
int height = int.Parse(frameListener.ReceiveFrame().ReadString());
int width = int.Parse(frameListener.ReceiveFrame().ReadString());
using (MemoryStream ms = new MemoryStream(bytebuffer)) //image data
{
Bitmap img = new Bitmap(Image.FromStream(ms), new Size(height, width));
pictureboxVideo.BeginInvoke((MethodInvoker)delegate{pictureboxbVideo.Image = img;});
}
I have a server & client model. The client is supposed to take a screenshot, and then send it to the server. The problem I'm having, is part of the screenshot is missing. What I mean by this is like 3/4 of the screen is black when opened in Paint or another app. When I send the screenshot command a second time, the file doesn't open at all, it's corrupt.
Here is my client side
if (plainText.Contains("screenshot"))
{
Bitmap bitmap = new Bitmap(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height);
Graphics graphics = Graphics.FromImage(bitmap);
graphics.CopyFromScreen(0, 0, 0, 0, bitmap.Size);
bitmap.Save("test.bmp");
writebuffer = File.ReadAllBytes("test.bmp");
stream.Write(writebuffer, 0, writebuffer.Length);
}
As you can see, it takes a screen shot, saves the image to a bitmap file, then reads the bytes into a buffer and sends it.
Here is my server side
foreach (user c in clientList)
{
if (c.RemoteIPAddress == currentRow)
{
NetworkStream stream = c.clientTCP.GetStream();
writeBuffer = Encoding.ASCII.GetBytes("screenshot");
stream.Write(writeBuffer, 0, writeBuffer.Length);
byte[] readBuffer = new byte[0];
readBuffer = new byte[c.clientTCP.ReceiveBufferSize];
int data = stream.Read(readBuffer, 0, readBuffer.Length);
string x = new Random().Next().ToString();
FileStream f = new FileStream(new Random().Next() + ".bmp", FileMode.Create, FileAccess.Write);
f.Write(readBuffer, 0, data);
f.Close();
Process.Start(x + ".bmp");
}
}
Here, I send the command to tell the client to send a screenshot, then receive the screen, and then write the said screenshot to a bitmap file.
I cannot seem to figure out what is causing the aforementioned issues in this code.
David's "answer" is useless. "hurr muh protocol". The reason your code isn't working is because the max size of a packet is 65535, and you're assuming the image isn't going to be any bigger than that - i.e you're only reading one packet. To read all the packets, use this loop.
while (stream.DataAvailable)
{
data = stream.Read(readBuffer, 0, readBuffer.Length);
f.Write(readBuffer, 0, data);
}
It will check if data is avaliable in the stream, and read it till there's nothing left.
I cannot seem to figure out what is causing the aforementioned issues in this code.
That's because you have nothing to compare it to. There are lots of correct ways to send an image and lots of correct ways to receive an image. But if you want your code to work, the sender and receiver have to agree on how to send an image. This is called a "protocol" and they should always be documented.
If you had a document for this protocol, it would specify how the sender indicates the size of the image. And it would specify how the receiver determines when it has the entire image. You could then check to make sure both the sender and receiver do what the protocol says. If they do, then it would be the protocol that was broken.
When you're using a network connection or file and you're not using an existing protocol or format, document the protocol or format you plan to use at the byte level. That way, it is possible to tell if the sender or receiver is broken by comparing its behavior to the behavior the protocol specifies.
Without a protocol, there's no way to say who's at fault, since there's no standard of correct behavior to compare them to.
Do not ever think that your protocol is too simple to document. If it's so simple, then documenting it should be simple. And if you think it's too complex to document, then you have no prayer of implementing it correctly.
By the way, it's possible to have a protocol for which your sending code is correct. But it's very difficult. It's impossible to have a protocol for which your receiving code is correct since it has literally no possible way to know if it has the whole file or not.
I'm building a client/server solution which needs to have screen sharing functionality. I have something already "working" but the problem is that it only works over internal network, because my methodology is not fast enough.
What I am basically doing is that the client makes a request for the server asking for a screen image each 5 seconds (for example). And this is the code which is processed once this requests are received:
private void GetImage(object networkstream)
{
NetworkStream network = (NetworkStream)networkstream;
Bitmap bitmap = new Bitmap(
SystemInformation.PrimaryMonitorSize.Width,
SystemInformation.PrimaryMonitorSize.Height);
Graphics g = Graphics.FromImage(bitmap);
g.CopyFromScreen(new Point(0, 0), new Point(0, 0), bitmap.Size);
g.Flush();
g.Dispose();
MemoryStream ms = new MemoryStream();
bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
bitmap.Dispose();
byte[] array = ms.ToArray();
network.Write(array, 0, array.Length);
network.Flush();
ms.Dispose();
}
What are best methods to do what I'm trying to? I need to get at least 0.2 FPS (refresh each 5 seconds) Obs.: I'm using Windows Forms and it is being done over sockets.
How do TeamViwer and .rdp files work?
You can send only difference betwen present and last image. Look here: Calculate image differences in C#
If it wont be fast enough, you can divide your screen into smallers, like 100x100 or 50x50 bitmaps, check if this area had changed and if yes just send it to client.
You need to optimize your protocol, here are some suggestions:
break the input image in segments, send segments instead of full screen
only send a segment if it's different from the previously sent version
use http notification type of communication where your viewer sends a request but only gets a response if the server received new segments from the presenter, possibly several grouped together.
compress the image data, don't transmit raw
give users the option to choose the level of compression to speed things up or to get a better image
I doubt this would be in your budget but you can also encode the stream as streaming video
What about using an existing implementation? Or learning from it?
http://cdot.senecac.on.ca/projects/vncsharp/
This might be a little of a long shot, but here goes: I have a WPF project and need to stream MJPEG video. The library at http://mjpeg.codeplex.com/ seems to be one of the few PnP options. It works great for viewing a single stream. But, when you try to switch from one URI, to a second URI the end result is that you get frames from BOTH streams interlaced into the same image object on my WPF page. Both streams are live, not just a cache of the previous stream.
No matter what I try it seems like the first stream will not go away and the stopstream method in the decoder doesn't do a damn thing other than set a boolean value.
Here's is the pseudo code for how I'm using the library. Am I doing something wrong?
button_click{
//Create new decoder instance
//Remove the previous image object from my WPF page
//Add a new image object to the WPF page
//Stop stream
//Set the event for a new frams
//Request the new stream with a new URI
}
I have written to the decoder author with no response. I'm hoping that someone else that has used this library will be able to shed light on this.
If you call StopStream(), wait a bit, and then call ParseStream again, it should shut down the first stream, and only display the second one.
The better alternative would be to only use a single instance of MjpegDecoder for each stream you would like to view.
Of course, if you aren't sure how it works, you can just download the code, and see how it works.
Although the MJPEGDecoder library is great, it unfortunately creates a WPF BitmapImage and a System.Drawing.Bitmap at each frame. This is way too much.
What we need is a byte array, which is platform independant. Then it is up to the UI to convert it to an actual Image object.
So I took the AForge.NET MJPEGStream.cs object and tweaked it a bit so it sends a byte array instead of a Bitmap.
MJPEGStream.cs is robust as hell and very fast. I am using it in production to stream up to 30 streams. It automatically stops and restarts the stream as the URI changes, retries by itself if the cam stops responding...
Please take this gist, then use it this way :
var stream = new MJPEGStream("http://webcam.st-malo.com/axis-cgi/mjpg/video.cgi?resolution=352x288");
stream.NewFrame += img => {
Dispatcher.BeginInvoke(
System.Windows.Threading.DispatcherPriority.Render,
new Action(() => {
var bmp = new BitmapImage();
bmp.BeginInit();
bmp.StreamSource = new MemoryStream(img);
bmp.EndInit();
bmp.Freeze();
pic.Source = bmp;
}));
};
stream.Start();
Of course, have look on the documentation to benefit all its features.
I believe with JPGs, the width and height information is stored within the first few bytes. What's the easiest way to get this information given an absolute URI?
First, you can request the first hundred bytes of an image using the Range header.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Headers.Set(HttpRequestHeader.UserAgent, "Range: bytes=0-100");
Next, you need to decode. The unix file command has a table of common formats, and the locations of key information. I'd suggest installing Cygwin and taking a look at /usr/share/file/magic.
For gif's and png's, you can easily get the image dimensions from the first 32 bytes. However, for JPEGs, #Andrew is correct in that you can't reliably get this information. You can figure out if it has a thumbnail, and the size of the thumbnail.
The get the actual jpeg size, you need to scan for the start of frame tag. Unfortunately, one can't reliably determine where this is going to be in advance, and a thumbnail could push it several thousand bytes into the file.
I'd recommend using the range request to get the first 32 bytes. This will let you determine the file type. After which, if it's a JPEG, then download the whole file, and use a library to get the size information.
I am a bit rusty at this, but with jpeg, it might not be as simple as it seems. Jpeg has a header within each segment of data which has its own height / width and resolution. jpeg was not designed to be streamed easily. You might need to read the entire image to find the width and height of each segment within the jpeg to get the entire width and height.
If you absolutely need to stream an image consider changing to another format which is easier to stream, jpeg is going to be tough.
You could do it if you can develop a server side program that would seek forward and read the header of each segment to compute the width and height of the segment.
It's a bit Heath Robinson, but since browsers seem to be able to do it perhaps you could automate IE to download the image within a webpage and then interrogate the browser's DOM to reveal the dimensions before the image has finished downloading?
Use this code:
public static async Task<Size> GetUrlImageSizeAsync(string url)
{
try
{
var request = WebRequest.Create(url);
request.Headers.Set(HttpRequestHeader.UserAgent, "Range: bytes=0-32");
var size = new Size();
using (var response = await request.GetResponseAsync())
{
var ms = new MemoryStream();
var stream = response.GetResponseStream();
stream.CopyTo(ms);
var img = Image.FromStream(ms);
size = img.Size;
}
return size;
}
catch
{
return new Size();
}
}