I was successful on training and evaluate networks based on the CIFAR-10 samples. I'm using my own images with specific size . The networks were trained with GPU and able to evaluate with CPU. However, I'm not able to evaluate it with with GPU. The evaluation is using C# EvalDLL Wrapper. The deviceID is change from -1 to 0 to indicate the GPU # as shown below:
model0.CreateNetwork(string.Format("modelPath=\"{0}\"", modelFilePath), deviceId: 0);
Did I missed something?
Can anyone run GPU on C# EvalDLL Wrapper program ?
I'm using binary version of the CNTK (not CPU_Only).
I'm able to run the EvalWrapper on a GPU, using exactly the code you posted below. I just tested via CsEvalClient:
When running my CNN through EvalWrapper on a GPU box, the logs say that it used the cuDNN convolution engine, which is GPU-only. I take that as a sign that it really ran on the GPU.
When running the code on a CPU-only box, it screams that it can't find a CUDA driver, as I would expect.
When switching back to deviceId:-1 and run on the CPU-only box, the logs say that is uses the GEMM convolution engine, and that engine is CPU-only.
What do you see in the logs when CNTK loads the model?
I'm not sure this is the right procedure. Here is what I did.
Instead of using the binary vision, install the source code follow the installation instruction and compile the source code.
Copy the DLLs from "x64.release" directory to the directory where the "exe" file located (flowing https://github.com/Microsoft/CNTK/issues/394).
Note: for multiple projects in solution, make sure the configuration is the same (eg. release mode at x64).
With GTX 960, the processing speed to from 770ms to 1 ms for an image.
Related
Context: I'm looking at the effects of down sampling, then up sampling video files. I'm using Media Foundation .NET to expose MF in C#. Program currently goes through following process:
Take a high res video and read in each frame (SourceReader & ReadSample)
Down sample using custom code that manipulates at the byte level
Write the down sampled data to a new, lower res video file (using SinkWriter)
Repeat for a range of resolutions supported by Media Foundation
Read down sampled videos back in and up sample to the next higher resolution in the list below, again using custom code that manipulates each byte
Write the new data to a higher res file (again using SinkWriter)
Resolutions I'm using are:
2560,1440
2346,1320
2134,1200
1920,1080
1706,960
1494,840
1280,720
1068,600
854,480
640,360
428,240
214,120
Current situation: This works almost perfectly. I run through the down sample process and have 11 down sampled video files (one at each resolution in the list above), plus the original 1440p video. I then read in each of those 11 videos and up sample. It works for 10 of them.
The problem: when I try to take the (1280,720) video to up sample to (1494,840), I get:
System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.'
... when I try to read in the first frame. I can't figure out why. The SourceReader configures fine (at least, no error returns). I do things like Marshal.Copy to get the sampled frame data into managed memory space, which I initially assumed was the problem. Code doesn't get that far though - just errors as soon as I try to read the first frame sample. ReadSample is in a Try...Catch block, but exception remains unhandled, so no other error information returned.
I don't want to just start pasting in unhelpful code, so please let me know what is useful to see and I'll add to the question. Most of the code has been taken from MS tutorials for SourceReader and SinkWriter. Also worth keeping in mind that this works in most situations, so code isn't 'broken' as such.
I've tried compiling in Release and Debug, x86 and x64. Also tried Suppressing JIT Optimisation in Visual Studio options.
Any ideas on where to look next?
Turns out this is a problem with the Media Foundation .NET interface, not the underlying MF framework. I built a small test program in C++ that implemented key parts of the code and it went through fine.
Not sure why Media Foundation .NET was causing problems, but the solution was just to set attribute:
MF_SOURCE_READER_ENABLE_ADVANCED_VIDEO_PROCESSING
rather than
MF_SOURCE_READER_ENABLE_VIDEO_PROCESSING
With advanced processing on, it behaves properly.
In my application, I have provision to run Python scripts through IronPython. In one of the requirement, the data from my C# application should be displayed in 3D using Visualization Toolkit. When I try to use VTK in a script and execute it, I get "ImportError: No module named vtk" error. I have searched for file named vtkCommonPython.pyd but cannot find it on my computer. Should Python be installed even after installing IronPython? What is the correct way to use VTK from IronPython?
In most cases, IronPython does not work with native/CPython modules like what vtkCommonPython.pyd seems to be. You will probably want to look at .NET bindings for VTK in order to use it from your IronPython scripts.
ActiViz.NET might be what you should be using. Wiki/Documentation, Download
Other modules containing only pure python should typically work fine. It's 'just' the native extensions that do not work as there is no easy (and built-in) way to make native code callable from .NET/IronPython using the same conventions as in CPython.
For additional info see the FAQ ('Compatibility') and the issue tracker.
You might also want to keep an eye on the recent effort to revive and port ironclad to IronPython 2.7. This project aims to provide the marshalling layer required to invoke CPython modules.
What do you think is the best approach to solve this issue. I have a C# application that receives a XML list of files. Based off this XML list many files (~10 megabyte size) get copied from one folder on our local SAN to another folder on the same local SAN. But all those file copies transfer via the C# Windows App that is actually doing the file copying. This takes 20 minutes per job. Any good ideas on how to dispatch a set of copy instructions to the SAN via C# but not have the local box be involved in the transfer?
Does the Mono codebase offer a way to SSH into a box and execute commands? This is my only idea so far to greatly reduce the execution time these jobs take.
UPDATE
The SAN is the Clariion NS 480
http://www.tech.proact.co.uk/emc/emc_celerra_ns-480_nas.htm
I assume this is a Linux or Unix operating system inside.
Trying to find some better technical specs.
You could run a process on the SAN that executes a specific script every x minutes. Create that script from your c# application.
Usually I would use SSH from the C# application in such a situation to issue the needed copy commands BUT from the link you posted none of the protocols listed supports such a mechanism... neither SSH not TELNET (which I would NOT recommend for security reasons) is listed as a supported protocol... without such a protocol your only option is to have your local machine involved in the copy process... I highly recommend checking back with the vendor about SSH support...
Has anybody used Mapnik for rendering Maps from .net c# environment? I would like to use it to render maps in a desktop application developed under .net4. I found it the best open source map renderer tool, much more better, than the well known sharpmap, dotspatial (...). The 'only' problem is that it was written in C++ and Python and does not have a trivial way to use it from C#.
Do you know any .net Mapnik wrappers? Do you know any sample code?
I've just published my attempts in C# bindings.
They are available at: http://sourceforge.net/projects/mapniknet/
Some good new about Mapnik-.Net integration:
"Another thing that will help Windows developers is a way to use Mapnik from .Net languages. Although a full .Net wrapper is too much work besides all the other issues, hopefully I can make a meaningful start and open the door for .Net developers to leverage Mapnik in their applications."
Source: Mapnik homepage http://mapnik.org/news/2011/jun/09/gsoc_2011_windows_build_system_introduction/
NET-Mapnik
You might want to give this a try. Its pretty recent and has a simple example with options to create vector tiles.
https://github.com/kernelsanders/NET-Mapnik
So, finally I hacked it. Not a nice solution, but it works at least. In nutshell it operates in the following way:
developed a python script, which takes parameters as command line arguments. (using python api of mapnik)
call this python script by shell fom c# code (Process)
after map image has been generated, I show it in SharpMap as a background layer
In case of any map refresh (zoom, pan) I regenerate 'mapnik map layer'. I have also some kind of cache, in which I can store generated maps, so I can make tricks by using earlier generated map images or pre-generate them (e.g. generate maps for the next zoom level when user uses zoom tool). I also use some asynchron calls to do this, so I can create 1-3 map images paralelly.
The weak part of this solution is the communication between python and c# modules. I could not find better than command line arguments + shell.
My solution works, mapnik is fast, can genarate my maps less than a sec.
If you have any advice on .net - python communication (NO ironpython, it is not suitable for manik api), pls write it here. Thx.
Do you know about TAO Project?
It is a .Net project that takes unmanaged opengle library to .Net Managed Form by calling DllIMport() in C#.
You must do it for Mapnik, simaliar to TAOFramework.net
Does anybody have a known reliable way to create a video from a series of image files? Before you mod me down for not searching for the answer before posting the question, and before you fire off a simple message like "use FFMPEG," read the rest of this message.
I'm trying to create a video, it doesn't matter too much what format as long as it's widely supported, from a series of images (.jpg, .bmp, etc.). My platform is Windows Server 2008, 64-bit. If I can make the video from within my C# program, that's great, but I'm not averse to writing a series of image files to a directory and then firing off an external program to make a video from those images.
The only constraints are: it must work on my Windows Server 2008 system, and be scriptable. That is, no GUI programs that require operator intervention.
I found a number of similar questions on StackOverflow, and have tried several of the solutions, all with varying degrees of frustration and none with anything like success.
FFMPEG looks like a great program. Maybe it is, on Linux. The two Windows builds I downloaded are broken. Given this command line:
ffmpeg -r 1 -f image2 -i jpeg\*.jpg video.avi
One of the builds reads the images and then crashes due to data execution prevention. The other reads the first file and then spits out an error message that says "cannot find suitable codec for file jpeg/image2.jpg". Helpful, that. In any case, FFMPEG looks like a non-starter under Windows.
One answer to a previous posting recommended Splicer . It looks like pretty good code. I compiled the samples and tried to run, but got some cryptic error message about a file not found. It looks like a COM class isn't registered. I suppose I need to install something (DirectShow, maybe, although I thought that was already installed?). Depending on what's required, I might have a difficult time justifying its installation on a server. ("What? Why do you need that on a server?")
Another answer suggested the AviFile library from Code Project. That looks simple enough: a wrapper around the Windows AviFile subsystem. Except that the AVI files the package creates appear to have all of the frames, but only the first frame shows when I play the AVI in Windows Media Player. Well, that and if you try to create a compressed video, the program throws an exception.
So, I'm left wondering if there is a good, reliable way to do what I want: on a Windows system, create an AVI or other common video file format from a series of images, either through a .NET API or using an external program. Any help?
After working with it a while and taking a look at x264 and VideoLan, I went back to Splicer. It turns out that the cryptic error message was due to an error in my code.
It looks like Splicer will do what I want: programmatically create videos from a series of images.
Thanks to all who responded.
You can use VideoLan and I'd recommend it.
I've had direct experience in a C# application with VideoLan doing these two things:
Embedding a VLC viewer in my C# application (there are 3-4 C# "wrappers" for the VLC veiwer).
Using vlc.exe in a separate Process by sending it command-line arguments.
The fact that VideoLan has a command-line interface is a great thing. And VLC supports a command-line option that disables any visual element; so the VLC GUI doesn't pop up and neither does a command-line window. Thus, in a C# application you can farm out the video-related work to the VLC client. C# has the Process class which can manage your vlc.exe instances for you. It ends up being a pretty neat solution.
The handling of video on Windows (encoding and playback) is a real quagmire. And .NET isn't going to help you at all - you're always going to be either PInvoking/COM-interoping native APIs or using a .NET wrapper that someone else has built to do the same.
Anyway, I can't say I've used it, but a good bet seems to be the x264 library that is part of the videolan project. That gives you a self-contained C library that you can PInvoke and you don't have to worry about installing and DirectShow garbage on your server. It'll generate H.264 format video which is playable in Flash as I understand it.
I've used mplayer's mencoder to create AVI (with codecs msmpeg4/mp3, readable by default with Windows Media Player) from TGA files on linux like this:
mencoder mf://*.tga -mf fps=25:type=tga \
-audiofile /tmp/test.mp3 -oac copy \
-of lavf -ovc lavc \
-lavcopts vcodec=msmpeg4v2:mbd=2:mv0:trell:cbp:last_pred=3:vbitrate=3000 \
-o /tmp/test.avi
I know mplayer does have a Windows version.
Instead of just going through all the hassle, you could coppy all the images that you want to be included in your video to the clipboard and have it imported into Windows Live Movie Maker (dynamically, of course!) and have it compile for you!
All of this CAN be done through code, but this way, you don't have to create it yourself. As you said that the platform was Windows, and ALL Windows comps have Windows Live Movie Maker installed, so it shouldn't be a problem for anyone using your application unless
a) They uninstalled Windows Live Movie Maker
b) They are using a computer running on a Windows Version more than 8 or 9 years old...