Develop Reference

Detecting truck wheels

I am currently working on a project which we have a set of photos of trucks going by a camera. I need to detect what type of truck it is (how many wheels it has). So I am using EMGU to try to detect this.
Problem I have is I cannot seem to be able to detect the wheels using EMGU's HoughCircle detection, it doesn't detect all the wheels and will also detect random circles in the foliage.
So I don't know what I should try next, I tried implementing SURF algo to match wheels between them but this does not seem to work either since they aren't exactly the same, is there a way I could implement a "loose" SURF algo?
This is what I start with.
This is what I get after the Hough Circle detection. Many erroneous detections, has some are not even close to having a circle and the back wheels are detected as a single one for some reason.
Would it be possible to either confirm that the detected circle are actually wheels using SURF and matching them between themselves? I am a bit lost on what I should do next, any help would be greatly appreciated.
(sorry for the bad English)
UPDATE
Here is what i did.
I used blob tracking to be able to find the blob in my set of photos. With this I effectively can locate the moving truck. Then i split the rectangle of the blob in two and take the lower half from there i know i get the zone that should contain the wheels which greatly increases the detection. I will then run a light intensity loose check on the wheels i get. Since they are in general more black i should get a decently low value for those and can discard anything that is too white, 180/255 and up. I also know that my circles radius cannot be greater than half the detection zone divided by half.

In this answer I describe an approach that was tested successfully with the following images:
The image processing pipeline begins by either downsampling the input image, or performing a color reduction operation to decrease the amount data (colors) in the image. This creates smaller groups of pixels to work with. I chose to downsample:
The 2nd stage of the pipeline performs a gaussian blur in order to smooth/blur the images:
Next, the images are ready to be thresholded, i.e binarized:
The 4th stage requires executing Hough Circles on the binarized image to locate the wheels:
The final stage of the pipeline would be to draw the circles that were found over the original image:
This approach is not a robust solution. It's meant only to inspire you to continue your search for answers.
I don't do C#, sorry. Good luck!

First, the wheels projections are ellipses and not circles. Second, some background gradient can easily produce circle-like object so there should be no surprise here. The problem with ellipses of course is that they have 5 DOF and not 3DOF as circles. Note thatfive dimensional Hough space becomes impractical. Some generalized Hough transforms can probably solve ellipse problem at the expense of a lot of additional false alarm (FA) circles. To counter FA you have to verify that they really are wheels that belong to a truck and nothing else.
You probably need to start with specifying your problem in terms of objects and backgrounds rather than wheel detection. This is important since objects would create a visual context to detect wheels and background analysis will show how easy would it be to segment a truck (object) on the first place. If camera is static one can use motion to detect background. If background is relatively uniform a gaussian mixture models of its colors may help to eliminate much of it.

I strongly suggest using:
http://cvlabwww.epfl.ch/~lepetit/papers/hinterstoisser_pami11.pdf
and the C# implementation:
https://github.com/dajuric/accord-net-extensions
(take a look at samples)
This algorithm can achieve real-time performance by using more than 2000 templates (20-30 fps) - so you can cover ellipse (projection) and circle shape cases.
You can modify hand tracking sample (FastTemplateMatchingDemo)
by putting your own binary templates (make them in Paint :-))
P.S:
To suppress false-positives some kind of tracking is also incorporated. The link to the library that I have posted also contains some tracking algortihms like: Discrete Kalman Filter and Particle Filter all with samples!
This library is still under development so there is possibility that something will not work.
Please do not hesitate sending me a message.

WPF: Finding 3D-visuals that are partially inside a 2D rectangle

I am making a WPF program with the possibility to modify data graphically in 3D. In order to give the user the option to select multiple graphical objects at the same time, I want to implement a selection rectangle. (Just like the one in windows explorer.) A common functionality in programs like this one is to have 2 different functions for the selection rectangle, and that the user can somehow choose which of the methods should be used.
Any object that is partially or completely inside the rectangle is selected.
Only objects that are completely inside the rectangle are selected.
The 2nd method is straight forward by using the bounding box of each object, and check if it is inside the rectangle. The 1st one on the other hand, seems to be quite some work. All my graphical objects are complicated 3D figures, and can be rotated by the user in any way. At the moment I am unable to find any other way than checking if any of the triangles in the mesh of any of the objects cross my 2D rectangle, and that can be quite time consuming.
I have little experience with WPF 3D, but I have done this before in OpenGL. Then I could tell OpenGL to draw a specific area of the screen, and the collect a list of objects that was visible in the specific area. All I needed to get the functionality I wanted was about 5 lines of code.
I guess my question is this:
Is there a way to do this with WPF 3D, similar to the OpenGL approach?
If not, is there any other smart way to find all objects (Visual3D) in a viewport that is partially behind a 2D rectangle?
I refuse to believe I am the only one with this kind of problem, so I hope a clever mind can point me in the right direction.
Regards,
Sverre
Thank you for your answer!
The 2D-rectangle is just in front of the camera and extending infinitely forward. I want to get any object that is partially or completely inside that frustum.
The camera we are using is an orthographic or perspective projection camera (System.Windows.Media.Media3D.ProjectionCamera). The reason we are not using the matrix camera is that we are using a 3rd party tool that does not support the matrix camera. But I am sure there is a way to get the matrix from a projection camera as well, so that is hopefully not the problem.
In theory your solution sounds like just what we need, but I am not sure how to proceed. Do you have any links to sample-code, or can you give some more hints on how to actually implement this?
Btw: Since we are working with WPF, we do not have direct access to DirectX. At least that’s what we have concluded after some research. You mention use of the z-buffer, which we haven’t been able to access through WPF. If you know a way to access the z-buffer, it’s greatly appreciated! This is of-topic, but we have struggled to disable the z-buffer for some time, but have given up…
Best regards,
Sverre

Is your intersection region a 2d rectangle or a frustrum based at a 2d rectangle and extending infinitely forward (or perhaps to some clipping limit)? If it can be construed as a viewing frustrum, then you can leverage the existing capabilities of the graphics system to render the scene using a Camera View and Projection that corresponds to your originating rectangle, with all lighting and shading disabled and colors chosen specifically to 'tag' the different objects in your scene. This means you can use the graphics hardware to perform the clipping/projection as a 'rendering' operation, then simply enumerate the pixel values as 'tags' to determine the objects present in the rectangular view.
If you need to restrict selection to an actual 2d slice (or a very shallow frustrum), you can use the Z-buffer (if you can get access to it) to exclude tagged pixels that are outside the Z range of your desired selection frustrum.
The nice thing about this approach is that you probably already have the Camera matrix (it's the same matrix used for your window for selection) and only need to change the Projection matrix to be a sub-set of the viewing window.

A 'smart' way would be to transform the rectangle into a box using the Camera's matrix
And then do a intersection of all the objects and the box.

.NET high level graphics library

I am programming various simulation tools in C#/.NET
What I am looking for is a high level visualization library; create a scene, with a camera with some standard controls, and render a few hunderd thousand spheres to it, or some wireframes. That kind of thing. If it takes more than one line to initialize a context, it deviates from my ideal.
Ive looked at slimDX, but its way lower level than im looking for (at least the documented parts, but I dont really care for any other). WPF perspective looked cool, but it seems targeted at static XAML defined scenes, and that doesnt really suit me either.
Basically, im looking for the kind of features languages like blitzbasic used to provide. Does that exist at all?

I'm also interested in this (as I'm also developing simulation tools) and ended up hacking together some stuff in XNA. It's definitely a lot more work than you've described, however. Note that anything you can do in WPF via XAML can also be done via code, as XAML is merely a representation of an object hierarchy and its relationships. I think that may be your best bet, though I don't have any metrics on what kind of performance you could expect with a few hundred thousand spheres (you're absolutely going to need some culling in that case and the culling itself may be expensive if you don't use optimizations like grid partitioning.)
EDIT: If you really need to support 100K entities and they can all be rendered as spheres, I would recommend that you bypass the 3d engine entirely and only use XNA for math. I would imagine an approach like the following:
Use XNA to set up Camera (View) and Perspective matrices. It has some handy Matrix static functions that make this easy.
Compute the Projection matrix and project all of your 'sphere' origin points to the viewing frustrum. This will give you X,Y screen coordinates and Z depth in the frustrum. You can either express this as 100K individual matrix multiplications or multiplication of the Projection matrix by a single 3 x 100K element matrix. In the former case, this is a great candidate for parallelism using the new .NET 4 Parallel functionality.
If you find that the 100K matrix multplications are a problem, you can reduce this significantly by performing culling of points before transformation if you know that only a small subset of them will be visible at a given time. For instance, you can invert the Projection matrix to find the bounds of your frustrum in your original space and create an axis-aligned bounding box for the frustrum. You can then exclude all points outside this box (simple comparison tests in X, Y and Z.) You only need to recompute this bounding box when the Projection matrix changes, so if it changes infrequently, this can be a reasonable optimization.
Once you have your transformed points, clip any outside the frustum (Z < 0, Z > maxDist, X<0, Y<0, X>width, Y>height). You can now render each point by drawing a filled circle, with its radius proportional to Z (Z=0 would have largest radius and Z=maxDist would probably fade to a single point.) If you want to provide a sense of shading/depth, you can render with a shaded brush to very loosely emulate lighting on spheres. This works because everything in your scene is a sphere and you're presumably not worried about things like shadows. All of this would be fairly easy to do in WPF (including the Shaded Brush), but be sure to use DrawingVisual classes and not framework elements. Also, you'll need to make sure you draw in the correct Z order, so it helps if you store the transformed points in a data structure that sorts as you add.
If you're still having performance problems, there are further optimizations you can pursue. For instance, if you know that only a subset of your points are moving, you can cache the transformed locations for the immobile points. It really depends on the nature of your data set and how it evolves.
Since your data set is so large, you might consider changing the way you visualize it. Instead of rendering 100K points, partition your working space into a volumetric grid and record the number (density) of points inside each grid cube. You can Project only the center of the grid and render it as a 'sphere' with some additional feedback (like color, opacity or brush texture) to indicate the point density. You can combine this technique with the traditional rendering approach, by rendering near points as 'spheres' and far points as 'cluster' objects with some brush patterning to match the density. One simple algorithm is to consider a bounding sphere around the camera; all points inside the sphere will be transformed normally; beyond the sphere, you will only render using the density grid.

Maybe the XNA Game studio is what you are looking for.
Also take a look at DirectX.

WPF perspective looked cool, but it seems targeted at static XAML defined scenes
Look again, WPF can be as dynamic as you will ever need.
You can write any WPF program, including 3D, totally without XAML.

Do you have to use C#/.Net or would MonoDevelop be good enough? I can recomend http://unity3d.com/ if you want a powerful 3D-engine.

Imposing limitations on 4-point image transformation

I'm working on a 4-point image transformation for an application. The user would either drag each corner around to create a valid quadrilateral shape, or use its "bounding box" to resize it vertically, horizontally, and proportionally.
So far, I have the image transformation part working. However, I'm having a hard time imposing limitations to it so that...
The vertices doesn't cross over each other. In another words, I don't allow the user to create an hour-glass shape. It must always be a quadrilateral.
Likewise, the angle between each corner must be greater than 0. Otherwise, the vertices will end up in a line.
It has a minimum size on this image. That is, each corner has to be a certain distance away from each other, and their opposing lines.
The user cannot "flip" the image to its backside. The 4 corners (p1, p2, p3, and p4) must appear in clockwise order.
Concave kite shapes and triangles are valid.
I was wondering if there was a formula or a paper regarding these issues. I do currently have the formulas for finding if 2 line segments intersect (and where), as well as finding the closest point on a line to another point. Most of my implementations hasn't been user-friendly as I liked, as the corners would jump all over the place when imposing restrictions.
P.S. I'm using C# for this project, with DirectX. The application is solely 2D, however.

My suggestion is not to impose restriction but just drawing say just a red wireframe polygon if for you it's unacceptable.
It's more or less like when programmers put restriction on form input fields e.g. begin date must come before end date and don't allow you to type in a date in the begin field that's after the date in the end field... forcing users to tab to end field first, then entering the end date, then tabbing back to the begin field to enter the correct begin date.
Just showing the fields in red is MUCH more usable... and actually requires less coding.
Trapping user mouse movements is rarely a sensible idea. If you can't apply what user is asking just make this evident instead of constraining the movement... may be the user is moving the first vertex and wants to move another vertex later to a position that will make your transform valid (exactly like the two date fields example).
If you really think you MUST prevent invalid positions at all times (and that for example will forbid the user from flipping the image even if your transform would allow a flipping operation without problems) then the simplest solution that comes to my mind is just having an isValid() function and just not moving the keypoint to the new position if it's invalid.
Having code that "slides" around invalid state space areas is IMO quite difficult to handle nicely, also because invalid areas are going to be quite close to "almost invalid" areas.
Even in case "sliding" is a requirement I'd probably go for implementing an implicit isValid() approach by doing a local search around the keypoint position to find what is the closest valid point to the point the user is asking for.
Explicitly computing all sliding possibilities would be a nightmare to get correct and also to maintain if you need to change something in the transformation algorithm (and consequently in what is valid and what is not).

When you start out, and each time the user moves a handle, determine the set of horizontal and vertical lines that the other handles cannot pass, and enforce those boundaries during a drag. This will solve your first problem.
For the second, when the mouse moves during a drag, calculate the distance between the cursor and each of the other 3 handles; if it is less than your defined minimum distance then move the handle in a circular path around the other handle(s).

XNA 2D mouse picking

I'm working on a simple 2D Real time strategy game using XNA. Right now I have reached the point where I need to be able to click on the sprite for a unit or building and be able to reference the object associated with that sprite.
From the research I have done over the last three days I have found many references on how to do "Mouse picking" in 3D which does not seem to apply to my situation.
I understand that another way to do this is to simply have an array of all "selectable" objects in the world and when the player clicks on a sprite it checks the mouse location against the locations of all the objects in the array. the problem I have with this approach is that it would become rather slow if the number of units and buildings grows to larger numbers. (it also does not seem very elegant) so what are some other ways I could do this. (Please note that I have also worked over the ideas of using a Hash table to associate the object with the sprite location, and using a 2 dimensional array where each location in the array represents one pixel in the world. once again they seem like rather clunky ways of doing things.)

For up to hundreds of units, it should be fast enough to simply do a linear search O(n) over all the units in the world if the click regions are circles or rectangles. Especially seeing as it will be once per click, not once per frame.
If your units are not circular or rectangular, check against a bounding circle or rectangle first, and if that passes check against the more complicated bounding shape.
For a more detailed answer, here's my answer to a similar question about space partitioning. There I mention bucketed grids and quadtrees as potential structures for performance optimisation.
But you should never do performance optimisation until you have tested and actually do have a performance problem!

If you have a class that manages drawabel objects you could have a static int that you increase every time you make a new object, and save the old one as a local instance of Color in the drawabel object. You can then use the .Net type converter to make its to bye arrays and back, dont remember its name and im on my phoneon a train so can't check for you im afraid.
When you build the color from the byte array just remember to max the alpha channel, and if you happen to get too many objects you might overrun the indexes you can use.. not sure what to do then... probably have all your objects reaquire new colors from 0:0:0:255 again since hopefully some old ones are no longer in use :P
Not sure i made alot of sense but since im on a train thats all i can give you, sorry :)

You could use pixel perfect picking, which scales very well to massive numbers of objects (and has the advantage of being pixel perfect).
Essentially you render your scene using a unique colour for each object. Then you resolve the backbuffer into a texture and get the texture data back, finally you can simply check the pixel underneath the mouse and find out which object the mouse is on.
You can be clever about the information you get back, you can request just the single pixel the mouse is on top of.
Color[] pixel = new Color[1];
texture.GetData(pixel, mousePosition.Y * texture.Width + mousePosition.x, 1);
//pixel[0] == colour of the item the mouse is over. You can now look this up in a dictionary<Color, item>
You should be careful not to stall the pipeline by doing this (causing the CPU to wait for the GPU to render things). The best way to do this is to swap between 2 render targets, and always GetData from the render target you used last frame, this means the data is a frame out of date, but no human has fast enough reactions to notice.
Addendum in response to your comment.
To assign a unique colour to each object, simply increment a byte for each object. When that byte overflows, increment another, and when that one overflows increment another; Then you can use those three bytes as Red, Green and Blue. Remeber to keep alpha at max value, you don't want any see through objects!
To resolve the backbuffer is slightly changed in XNA4. Now you must render to a rendertarget and resolve that. To do this is pretty simple, and outlined by Shawn Hargreaves here