My current project has required me to learn face detection/tracking and image processing, given my experience in c#, I chose Emgu CV as my choice library for face detection and tracking. From what I've learned so far, I can do face detection and tracking, and basic image processing.
My goal is to be able to place virtual hair on the detected face. What I want to achieve is similar to [this video]: http://www.youtube.com/watch?v=BdPmECfUFcI.
What I would like to know is the technique(s) to use in handling hair placement for different kind of hairstyles on the detected face. In what image format do I store the the hair?
After watching the video I noticed it considers the head as a flat rectangle and not as a rectangular prism (the 3D object), so it doesn't consider the use of perspective transformations and I will not consider it too. This is a limitation but serves as a decent first step in doing such placements. Note that it is not a simply matter of taking perspective into consideration, your face tracking algorithm also needs to be able to handle more complicated configurations (the eyes might not be fully visible, for example).
So, the first thing you want is a bounding rectangle aligned according to the angle the eyes make with the x axis, illustrated in the following right figure (the red segment indicates the connection between the eyes). The left figure shows a typical bounding box aligned to the axis, which doesn't serve for this problem.
The problem is also simplified after you consider the head is symmetric, so you know the top middle point in the above figure is the middle of the top of your head. Also, considering that a typical head will likely be larger at top than at bottom, then you have something like in the following figure where the width of the rectangle is close to the width of the forehead. You could also consider a bounding rectangle on only upper half of the head, for example.
Now all that is left is positioning some object in this rectangle. For that, you need to augment the description of this object to be positioned so it is not purely pixels. We can define "entrance width" (EW) and "entrance middle point" (EM). This EW establishes the width needed in the other rectangle (the head one) to position it. So, if EW is smaller than the needed value, you upscale this object, respectively for when EW is larger. Note that the full width of the head's rectangle is usually an overestimation to position this object, so you can experiment with percentages of the width. The EM value is useful to know how you will position this object over the head. In the following figure, EW is the horizontal blue dashed horizontal, and EM is the middle point on it. The vertical blue line indicates how much over the EM you want to move this object inside the top segment of head's rectangle.
The only other special thing this object needs is a value that is considered as background. So when painting this object it is easy to know whether to make a point fully transparent (the background value) or fully opaque (anything else). This was the sketch I had in mind of what needs to be basically done.
Related
This question already has answers here:
Non-Affine image transformations in .NET
(3 answers)
Closed 1 year ago.
I need to combine two images in C# ( 4.7.2 ), and have the top image transformed putting each of the four corners at specific coordinates in the image.
Is that possible? Preferably with a solution that doesn't require spending a ton of money. As far as I can tell i can't do it with the Bitmap/Graphics classes.
Image of what I'm trying to do
Shear (or skew), which is what an affine transform such as used in GDI+ or WPF, is unlikely to do what you want, if I understand the question correctly. With shear/skew the transformed coordinate space is still a parallelogram, whereas in your image, the original rectangle is squeezed or stretched arbitrarily.
Assuming that's correct, I would recommend using the features in the WPF Media3D namespace (WPF, simply because it's the most accessible 3D API in the .NET context). In particular, you will want to define a texture that is your original bitmap. Then you will want to define a quadrilateral 2D surface in 3D coordinate space with sufficient resolution (i.e. triangles) for your purposes (see below), and where the triangles in that surface are constructed by tessellating the shape that you want as your final image, and where you've interpolated the texture (UV) coordinates for that shape across the vertexes that result from the tessllation.
How many triangles you actually want depends on the desired quality. In theory, you could use just two. This is the simplest approach, and determining the UV coordinates is trivial, because you only have your original four corners. But there will be a visual discontinuity along the diagonal where the two triangles meet, where the interpolation of the texture pixels changes direction due to the triangles not being square to each other.
For better results, you'll need to use more triangles. But then this complicates the assignment of the UV coordinates. For each inner vertex of this surface, you'll need to interpolate across the surface. This is probably easier to do if you generate the tessellation in the first place by subdividing the quadrilateral with lines connecting opposite sides (which will form smaller interior quadrilaterals bounded by intersecting lines) and then just divide each of those quadrilaterals into pairs of triangles. If you do it this way, then you can use the distance along each line to determine the appropriate U or V coordinate at each vertex that line goes through.
Having created the appropriate texture and geometry, it's a simple matter to render the result into a RenderTargetBitmap via the Viewport3DVisual class, and then do whatever you want with that bitmap.
Now, all that said…
If it turns out that your problem can be simplified such that shear/skew is sufficient for your needs, you can look at De-skew characters in binary image for help with that. In that particular example, they are trying to undo skew caused by optical effects, but skewing is skewing; the same exact principle works in either direction.
Even if your problem is not amenable to shear/skew approaches, before you implement your own solution (e.g. based on my outline above), you may want to look at other available tools. Information about some options can be found in, for example, Image Modification (cropping and de-skewing) in C# and Image comparison - rotation, alignment and scaling.
I am trying to segment arms from a Kinect depth image in my app (click for larger picture):
I tried using joint positions to get the vector between elbow and wrist/hand-tip, and created a 2D bounding rotated rectangle between these two joints, and then removed all pixels outside the rectangle. The problem is that, depending on the distance from the sensor, this rectangle changes width, and can become trapezoidal (e.g. if hand is closer to the camera), so it can basically only allow me to discard parts of the image before doing actual processing.
When the hand is near the body (like my left arm below), I need to detect the edge of the hand - presumably by checking the depth gradient. But I couldn't find a flood fill algorithm which "stops" at gradients.
Is there a better approach perhaps? I could use an algorithm idea.
Me and some guys are creating a simple game in XNA 4.0 (yea, i know it's not supported by MS anymore, but it's requirement given by our tutors). Recently I wrote Light-Pre Pass Renderer based on J. Coluna's one. It was working fine until we added some meshes with bump and albedo maps. Now we've got strange bug. Here are given the examples:
I dont't have a clue what causes these artifacts (green/purple). Sometimes similar artifacts occur on the floor and they are black. Do you have any idea what might be the problem in renderer?
If my post isn't clear enough let me know, I'll try clarify it.
You have not provided any code so I have to base my answer on observation only.
I believe the problem is from flipped normals related to how the object geometry was created.
If you take a close look on image #3 - you will notice that the purple artifact can be seen on the left side of the object as well but in a narrower area. according to that this is the theory I offer:
Your animator created the object and didn't like the sharp edges - so to get rid of them he or she rotated the edge vertexes and moved them inner to in a manner that overlap the shape exterior.
If I'll try to illustrate it somehow it would look something like this:
original object:
+-------+
|=====|
|=====|
|=====|
+-------+
vs. tinkered object:
-------
|x===x|
|=====|
|x===x|
-------
Where the '+' is converted to 'x' - meaning the vertex was rotated and moved further inside the shape. this probably inverted the normals which affect the light being reflected back from the object.
The reason we see in image #3 a narrow area with artifact in the left is probably since the artist rotated all corners at the same time - and if that is the case I believe the if you rotate the shape the phenomena will be symmetric meaning you will see again wider artifact to the right of the object and narrower to the left - but if you flip the shape (rotate on Y 180 degrees) the phenomena will flip with it.
Another option to test for this will be to put a new simple box shape into the scene and check if the artifact is gone.
I have some images that I'd like to draw a polyon around the outer edges. The images themselves are on transparent backgrounds and I've created an array of the pixels in the images which contain a point and are not transparent (or white).
Now, my question is: how do I draw an accurate polygon around the outer edge points? I've used a Graham Scan algorithm that I read about to create a convex hull around the edges but this doesn't seem to work for objects with concavities. For example:
http://i48.tinypic.com/4s0lna.png
The image on the left gets filled in using this method with the one on the right. As you can see, it's 'filling in' a little too much.
I assume there must be some other algorithm or approach that can be used to solve this, but I'm not sure of where to look or what it might be called. Could anyone point me in the right direction? I'm using C#/.net and hopefully there might be something that already exists which could work along these lines.
I think the 2D "Alpha shapes" algorithm would the right choice for you.
http://www.cgal.org/Manual/latest/doc_html/cgal_manual/Alpha_shapes_2/Chapter_main.html
Alpha shapes can be considered as a generalization for the "convex Hull" algorithm that allows for generation of more general shapes.
By using alpha shapes you will be having control over the level of details to be captured by the resultant shape by changing the alpha parameter value.
You can try the java applet here : http://cgm.cs.mcgill.ca/~godfried/teaching/projects97/belair/alpha.html
to have better understanding about does this algorithm do.
You can start on a pixel by pixel level, using a flood-fill approach.
Start in the corner, checking that it does have zero alpha.
Check the neighbours for zero alpha and iterate until we have no unchecked neighhours.
This gives you a mask for the image which will consist of two simply connected regions, the interior and exterior.
The set you seek then consists of:
all the points in the exterior which are on the boundary of the interior.
You can then turn that into a polygon by:
Take an initial polygon that consists of all the points in the edge set
Remove redundant vertices that lie along straight edges.
Here's the setup: This is for an ecommerce art site where some paintings are canvas transfers. The painting wraps around the sides and top and bottom of the canvas. We have high-res images of the entire painting, but what we want to display is a quasi-3D representation of the image in which you can see how the sides of the painting wrap around the canvas. Here's a rough sketch of what I'm talking about:
My question is, how can I rotate an image in 3D space? The approach I think I'd like to take, is to cut off a portion of the top and side of the image, and rotate then in 3D and then stich it back on to the top and side to give it the 3D look. How do I go about about doing that? It can be done using any .Net technology (GDI+, WPF etc.).
In WPF using the ViewPort3D class you can create a cuboid which is 8x5x1 units. Create the image as a texture and then apply the texture to the front face (8x5) and the side faces (5x1) and the top and bottom faces (8x1) using texture coordinates. The front face coordinates should be: (1/9, 1/6), (8/9, 1/6), (1/9, 5/6) and (8/9, 5/6) for the front face, and from the nearest edge to those coordinates for the sides, e.g. for the left side: (0, 1/6), (1/9, 1/6), (0, 5/6) and (1/9, 5/6) for the left side.
Edit:
If you then want to be able to perform rotations on the 3D canvas model you can follow the advice here:
How can I do 3D transformation in WPF?
It looks like you're not needing to do real 3D, but only needing to fake it.
Chop off four strips along the top, bottom, left and right of the image. Toss the bottom and right (going by your sketch in the question). Scale and shear the strips (I'm not expert enough at .net/wpf to know how, but it can do it). The top would be scaled vertically by a factor of 0.5 (a guess - choose to fit the desired final 3D-looking image) and sheared horizontally. The result is composited onto the output image as the top side of the canvas. The left strip would be scaled horizontally and sheared vertically.
If the end user is to view the 3D canvas from different angles interactively, this method is probably faster than rendering an honest 3D model, which would have to do texture mapping and rasterizing the model into a final image, which amounts to doing the same math. The fun part is figuring out how to adjust the scaling and shearing parameters.
This page might be educational: http://www.idomaths.com/linear_transformation.php
and this could be useful reference http://en.csharp-online.net/GDIplus_Graphics_Transformation%E2%80%94Image_Transformation
I dont have any experience in this kind of stuff. But when i saw this question, the first thing comes to my mind is the funny Unicornify for SO.
In this making of article by balpha, he explained how the 2d unicorn sphere is rotated in 3d space.
But the code is written in python. If you are interested, you can take a look into that. But am not exactly sure this would help you.
The brute force approach (which might be the easiest approach), is to map the u,v texture coordinates for each of the three faces, onto three billboards representing three sides of the canvas (a billboard is just two triangles that make a rectangle). Then, rotate the whole canvas (all three billboards) using matrix transforms. Tada!
Alternately, you can move the 3-space camera position with a transform, rather than the canvas. Six of one, half the other - as they say.