I'm writing a Windows Phone 7 app that deals with a lot of images - These images can range from a few hundred pixels up to 1080P (Potentially higher in future).
Images are very resource intensive so I've gone down the path of caching + resizing images on the phone before displaying them.
This means on first time setup after a user has entered the IP address of the image store I can ask them to wait a few minutes while it's all retrieved/resized/cached. From then on they can have nice and snappy performance.
At the moment my cache manager tracks Images by a dictionary of Uri's and file locations. I have a Queue that processes up to 5 images at a time (Async web requests, resizing is semi done on thread pool thread).
The problem I have is that the WritableBitmap class in Silverlight is a UI element, meaning I have to transition to the UI thread via the Dispatcher to do the actual resizing which is stupid and slows the whole thing down - It also means my Cache Manager is effectively single threaded.
So it goes Cache Manager (Thread Pool) -> Async Web request (Thread Pool) -> Callback (Thread Pool) -> Resizing (UI Thread) -> Marking cache job as complete (Thread Pool).
I've been looking for a 3rd party library that will A) Compile and run on Windows Phone 7 and B) Be able to resize images of various formats by manipulating a stream or byte array and not be dependent on the UI thread.
Any one had any experience with this?
Cheers,
Tyler
In order to reduce the download size and in order to remove the burden of processing from the phone CPU, then I'd push this work out to a web service.
For example, you could host a service like the open source WebImageResizer code somewhere online http://webimageresizer.codeplex.com/ - e.g. on a free AppHarbor server.
Or you could use a commercial (freemium) service like:
http://webresizer.com/app/
http://www.appspotimage.com/
Either of these would enable you to rapidly process the images on a server with a superfast connection and to juse deliver smaller images to the phone with its limited data connection.
The ImageTools library both supports WP7 and suports image resizing, so you might have more success using this.
Try http://writeablebitmapex.codeplex.com/ created by Réne Schulte. It has a way better performance as the WriteableBitmap that is shipped with the SDK.
Are files you want to display on your server?
If yes, in my opinion you have chosen the wrong approach to this problem. It's pointless to transfer heavy images, then resize them. I recommend keeping them in either low and high resolution on the server.
If not, I can't help you. I found this article but I think you have already seen it.
Tyler - You mention that on initial launch, the application does retrieve the images from an IP address (on the internet I presume)?
This means on first time setup after a user has entered the IP address of the image store I can ask them to wait a few minutes while it's all retrieved/resized/cached. From then on they can have nice and snappy performance.
If that is correct, what I think some of the answers have suggested in creating a proxy is actually a feasible solution. I would structure it like this:
Your App -> Web Request to Proxy Handler
string imageUrl = HttpUtility.UrlEncode("http://[user's_ip]/path_to_image.png");
http://domain.com/your_proxy.ashx?users_image_url=imageUrl
Your proxy handler on the back end should request the image, resize it, and return the resized image back to your application.
Your App -> Cache the Returned Resized Image
Loop as needed...
UPDATED:
It turns out there is a way to resize images on WP7 outside of the UI thread using the native WriteableBitmap class. There is a method called .SaveJpeg() that allows this, and we're using it currently in our WP7 application CitySourced. Your code should look something like this:
wb.SaveJpeg(stream, width, height, orientation, quality);
The only bummer is that it only works by writing a .JPG file and there's no .PNG support. Let me know if this works for your use case.
I don't think your problem is performance so much but how you are going about displaying the images. I think it would be better to always showing a default thumbnail and then update the thumbnail once it has been resized by your background job. The app should be usable much more quickly.
http://www.dtksoft.com/dtkimage.php
This works with all file formats, and is compatible with windows Mobile
Related
I have the following scenario in mind:
I want to send (via serial port) some commands to a device. This device does send me back a continuous stream of data (max. 12000 values per second).
To control some settings I need some buttons to send commands to the device to start/stop/change settings before and during data stream. Also I want to have a real time plot of this data. I will filter this data of course. Also at certain timestamps there will be a signal which indicates that I want to cut out a certain window of the received data.
This means I will have two charts. I made already some progress using WPF but now when I interact (zoom/pan) with the lower chart, the upper one freezes noticeable. This is because both have do be refreshed very often!
Work (data receiving/filtering) is done using threads but the update of the plot has to be done within the ui thread.
Any ideas how to solve this issue? Maybe using multiple processes?
You should use Reactive Extensions. It was built for this kind of thing.
http://msdn.microsoft.com/en-us/data/gg577609.aspx
Requesting a clear, picturesque explanation of Reactive Extensions (RX)?
On this second link, although the topic is javascript, much of what it says is about Reactive Extensions and cross-applies to Rx in C#.
I'm making a similar WPF application with real-time waveforms (about 500Hz). I have a background threads that receives real-time data, a separate threads to process them and prepare the data for drawing (I have a buffer with the "size" of the screen where I put the prepared values). In the UI thread I draw the waveforms to the RenderTargetBitmap which is in the end is rendered to the Canvas. This technique allows me have a lot of real-time waveforms on the screen and have zoom and pan working without any problems (about 40-50 fps).
Please let me know if you need some technical details, I can later share them with you.
I think you have some code in the UI thread that is not optimized well or can be moved to the background thread.
Btw, do you use any framework for charts?
Edit
philologon is right, you should use Rx for real-time data, it simplifies code A LOT. I also use them in my project.
Its a commercial product but there is a real-time WPF chart which can handle this use-case and then some. Please take a look at the Tutorial below:
http://www.scichart.com/synchronizing-chartmodifier-mouse-events-across-charts/
There is a live Silverlight demo of this behaviour here:
Sync Multichart Mouse Silverlight Demo
And this chart should be able to handle zooming while inputting values at high speed:
Realtime Performance Demo
Disclosure: I am the owner and tech-lead of SciChart
I am writing Metro application for image sharing (kind of). I have separate project for downloading and handling communication to/from server (running on separate thread). I am now wondering what classes should I use for storing image data. I remeber good old days when there was Image class just for this purpose. But today I can seem to find anything like it. There are classes like BitmapSource and DrawingImage, but those are UI classes and can not/should not be used in non-UI threads. What would you suggest me for storing image data and easily transferin it back to UI thread, so I can use it as ImageSource (preferably).
EDIT: I would like to keep image data in memory, if possible (not save to galery/remove on exit)
EDIT: So far I am using IRandomAccessStream, but I dont like this solution, because you can read only once from it (then reset must be performed). Further more - i am not sure how it behaves when multiple BitmapImage(s) have it set as source..
See http://winrtxamltoolkit.codeplex.com/ - the WriteableBitmap may be the ticket. Haven't played with it, but the overview seems in the ball park.
As you might surmise from the question title, we are required to decode and display multiple (e.g., eight) H.264 encoded videos at the same time (and keep them all time synchronized, but that's another question for another time). The videos are usually at at 25 FPS with a resolution of 640x480.
I'm going to provide a bit of background before I get to the crux of the problem.
The feature needs to be baked into a fairly large C# 3.5 (WinForms) application. The videos will occupy rectangles in the application - the managed code needs to be able to specify where each video is drawn as well as it's size.
We get the H264 packets in C# and fire them into a native H264 decoder to get YUV12 image data.
An early attempt consisted of converting the YUV12 images to RGB24 and BitBlt'ing them to a HWND passed into the native code from C#. While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos (on a 2.6 GHZ core 2 duo).
The current attempt spins up one-thread-per-cpu-core on startup and load balances the decoding/displaying of videos across these threads. The performance of this is mind-blasting (I find watching task manager much more interesting than the videos being displayed). UI-wise, it leaves a lot to be desired.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
Gah! CanOfWorms.Open().
Problem: When the C# application receives focus, it jumps to the front of the Z-Order and hides the video windows.
Solution: Place the video windows Always On Top.
Problem: When the user switches to another application, the video windows are still on top.
Solution: Detect activation and deactivation of the C# application and show/hide the video windows accordingly.
Problem: User says, "I want my videos playing on one monitor while I edit a Word document in the other!"
Solution: Tell user to shut up and that Word sucks anyways.
Problem: I get fired.
etc. etc.
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Any thoughts/suggestions? Am I completely out to lunch here?
I'm more than open to taking a completely different approach if one exists (This project required a lot of learning - winning the lottery would have a greater likelihood than me having picked the best approach at every step along the road).
Forget about BitBlt-ing and do this:
for each window you want your video to be played, create one DirectShow graph and attach the renderer of the graph to that window
before renderer in the graph put the samplegrabber filter. It will allow you to have callback in which you'll be able to fill the buffer
instead of blitting, decode to the buffer provided in samplegrabber.
In addition, I guess that you'll be able to put raw YUV12 into the buffer, as VMRenderer is able to display them directly.
Use DirectShowNet library.
EDIT:
And yes, BTW, if the videos are on the same 'canvas', you can use same technique with renderer and create only one large window, then shift decoded video rectangles 'by hand' and put them into the framebuffers buffer.
YET ANOTHER EDIT:
BitBlts are ALWAYS serialized, i.e. they can't run in parallel.
The millisecond we started drawing to an HWND created on the UI thread (e.g., a panel docked in a WinForms control) from a non-UI thread, we started getting all sorts of funky behavior due to the un-thread-safeness of WinForms. This led us to create the HWND's in native code and draw to those, with C# providing the rectangles they should be drawn to in screen coordinates.
What kind of funky behavior?
If you mean flickering or drawing delay, have you tried to lock() the panel or any other class for thread/drawing synchronisation?
Again: Whats the exact problem when you send the data to the decoder, receive a image, convert it and then draw it with an OnPaint handler. (Setup a different thread that loops at 25fps, call panel1.Invalidate() then)
I guess the crux of the problem is that we have HWND's created on a non-UI thread and we want to 'simulate' those being embedded in the C# application.
Don't do that. Try to draw the received data in your c# application.
In general, I wouldn't reccomend mixing native code and c#. Having the h264 decoder in native code is the only exception here.
Use your threads to decode the video packets (as you already do) then have one thread that loops and calls Invalidate(as said above). Then have an OnPaint handler for each panel you are displaying a video in. In this handler get the most recent video picture and draw it (e.Graphics).
I hope this helps, but also need more information about the problem...
I like the DirectShow answer posted earlier, but I wanted to include an additional option that might be easier for you to implement, based on this excerpt from your question:
While functional, all BitBlt'ing had to happen on the UI thread which caused it to bog down when displaying more than a couple videos
My idea is to start from that code, and use the Async CTP for Visual Studio 2010 that is currently available and includes a go-live license. From there it should be a relatively simple to modify this existing code to be more responsive: just add await and async keywords in a few places and the rest of the code should be largely unchanged.
I want to create a simple video renderer to play around, and do stuff like creating what would be a mobile OS just for fun. My father told me that in the very first computers, you would edit a specific memory address and the screen would update. I would like to simulate this inside a window in Windows. Is there any way I can do this with C#?
This used to be done because you could get direct access to the video buffer. This is typically not available with today's systems, as the video memory is managed by the video driver and OS. Further, there really isn't a 1:1 mapping of video memory buffer and what is displayed anymore. With so much memory available, it became possible to have multiple buffers and switch between them. The currently displayed buffer is called the "front buffer" and other, non-displayed buffers are called "back buffers" (for more, see https://en.wikipedia.org/wiki/Multiple_buffering). We typically write to back buffers and then have the video system update the front buffer for us. This provides smooth updates, as the video driver synchronizes the update with the scan rate of the monitor.
To write to back buffers using C#, my favorite technique is to use the WPF WritableBitmap. I've also used the System.Drawing.Bitmap to update the screen by writing pixels to it via LockBits.
It's a full featured topic that's outside the scope (it won't fit, not that i won't ramble about it for hours :-) of this answer..but this should get you started with drawing in C#
http://www.geekpedia.com/tutorial50_Drawing-with-Csharp.html
Things have come a bit from the old days of direct memory manipulation..although everything is still tied to pixels.
Edit: Oh, and if you run into flickering problems and get stuck, drop me a line and i'll send you a DoubleBuffered panel to paint with.
It's hard to put this into the title, so let me explain.
I have an application that uses Direct3D to display some mesh and directshow(vmr9 + allocator) to play some video, and then send the video frame as texture to the Direct3D portion to be applied onto the mesh. The application needs to run 24/7. At least it's allowed to be restarted every 24hours but not more frequent than that.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video. But the Direct3D portion is still running fine, mesh still displayed. Once the application is restarted, everything back to normal.
So, I'm thinking of splitting the 2 parts into 2 different process. So that when ever the video process failed to play video, at least I could restart it immediately, without loosing the Direct3D portion.
So here comes the actual question, whether it's possible to pass the texture from the video player to the direct3d process by passing the pointer, aka retrieve the texture of another process from pointer? My initial guess is not possible due to protected memory addressing.
I have TCP communication setup on both process, and let's not worry about communicating the pointer at this point.
This might be a crazy idea, but it will work wonder of it's ever possible
Yes you can do this with Direct3D 9Ex. This only works with Vista and you must use a Direct3DDevice9Ex. You can read about sharing resources here.
Now the problem is that directshow seems to be giving problem after a few hours of playback, either due to the codec, video driver or video file itself. At which point the application simply refuse playing anymore video.
Why not just fix this bug instead?
If you separate it out as a separate process then I suspect this would not be possible, but if it were a child thread then they would have shared memory addressing I believe.
Passing textures doesn't work.
I'd do it using the following methods:
Replace the VMR with a custom renderer+allocator that places the picture into memory
You allocate memory for pictures from a shared memory pool
Once you receive another picture you signal an event
The Direct3D process waits for this event and updates the mesh with the new texture
Note you'll need to transfer the picture data to the graphics card. The big difference is that this transfer now happens in the Direct3D app and not in the DirectShow app.
You could also try to use the VMR for this. I'm not sure if the custom allocator/renderer parts will allow you to render into shared memory.
Maybe you could use the Sample Grabber in your DirectShow host process to get the image as a system memory buffer. Then you could use WriteProcessMemory to write the data into a pre-agreed address (which you setup over TCP or something) in your Direct3D app.