Related
Need help because I'm new to OpenGL.
So, my task is to create a control that will draw in real time the process of cutting a workpiece on a CNC machine.
I tried to do it classically through glBegin, glEnd and everything works fine, but due to the large number of vertices, it starts to work slowly at the end. Therefore, I decided to try using VertexBufferArray () and VertexBuffer () - this also works, but in this situation I do not understand how to change the line width and its type (specifically, I have two types - a regular line and a dash-dotted line).
Here is my method where i use array
private void CreateBufferAndDraw(OpenGL GL)
{
try
{
if (shaderProgram == null && vertexBufferArray == null)
{
var vertexShaderSource = ManifestResourceLoader.LoadTextFile("vertex_shader.glsl");
var fragmentShaderSource = ManifestResourceLoader.LoadTextFile("fragment_shader.glsl");
shaderProgram = new ShaderProgram();
shaderProgram.Create(gL, vertexShaderSource, fragmentShaderSource, null);
attribute_vpos = (uint)gL.GetAttribLocation(shaderProgram.ShaderProgramObject, "vPosition");
attribute_vcol = (uint)gL.GetAttribLocation(shaderProgram.ShaderProgramObject, "vColor");
shaderProgram.AssertValid(gL);
}
if (_data == null)
{
_data = new vec3[_Points.Count];
}
else
{
if (_data.Length != _Points.Count)
{
_data = (vec3[])ResizeArray(_data, _Points.Count);
}
}
if (_dataColor == null)
{
_dataColor = new vec3[_Points.Count];
}
else
{
if (_dataColor.Length != _Points.Count)
{
_dataColor = (vec3[])ResizeArray(_dataColor, _Points.Count);
}
}
for (int i = _dataTail; i < _Points.Count; i++)
{
_data[i].y = _Points[i].Y;
_data[i].x = _Points[i].X;
_data[i].z = _Points[i].Z;
_dataColor[i] = new vec3(_Points[i].ToolColor.R / 255.0f, _Points[i].ToolColor.G / 255.0f, _Points[i].ToolColor.B / 255.0f);
}
_dataTail = _Points.Count;
// Create the vertex array object.
vertexBufferArray = new VertexBufferArray();
vertexBufferArray.Create(GL);
vertexBufferArray.Bind(GL);
// Create a vertex buffer for the vertex data.
var vertexDataBuffer = new VertexBuffer();
vertexDataBuffer.Create(GL);
vertexDataBuffer.Bind(GL);
vertexDataBuffer.SetData(GL, attribute_vpos, _data, false, 3);
// Now do the same for the colour data.
var colourDataBuffer = new VertexBuffer();
colourDataBuffer.Create(GL);
colourDataBuffer.Bind(GL);
colourDataBuffer.SetData(GL, attribute_vcol, _dataColor, false, 3);
// Unbind the vertex array, we've finished specifying data for it.
vertexBufferArray.Unbind(GL);
// Bind the shader, set the matrices.
shaderProgram.Bind(GL);
// Set matrixs for shader program
float rads = (90.0f / 360.0f) * (float)Math.PI * 2.0f;
mat4 _mviewdata = glm.translate(new mat4(1f), new vec3(_track_X, _track_Y, _track_Z));
mat4 _projectionMatrix = glm.perspective(rads, (float)this.trackgl_control.Height / (float)this.trackgl_control.Width, 0.001f, 1000f);
mat4 _modelMatrix = glm.lookAt(new vec3(0, 0, -1), new vec3(0, 0, 0), new vec3(0, -1, -1));
shaderProgram.SetUniformMatrix4(GL, "projectionMatrix", _projectionMatrix.to_array());
shaderProgram.SetUniformMatrix4(GL, "viewMatrix", _modelMatrix.to_array());
shaderProgram.SetUniformMatrix4(GL, "modelMatrix", _mviewdata.to_array());
// Bind the out vertex array.
vertexBufferArray.Bind(GL);
GL.LineWidth(5.0f);
// Draw the square.
GL.DrawArrays(OpenGL.GL_LINE_STRIP, 0, _dataTail);
// Unbind our vertex array and shader.
vertexBufferArray.Unbind(GL);
shaderProgram.Unbind(GL);
}
catch (Exception ex) { MessageBox.Show("CreateAndPlotData" + "\n" + ex.ToString()); }
}
This is what i get
As you see, all lines have the same width. So, my question is: Does anyone know what i can do about this?
And another one: what if i need to show a point of the current location of the instrument? I should create another array with one Vertex ?
p.s. Sry for my english, and here here is picture of what i get with glBegin/glEnd
No tag spamming intended.
I'm doing this project partially guided by tutorials in C ++ and C #. Therefore, if someone knows how to do this in C ++, then in this case I will just try to implement the same in C#.
I'm adding a feature to a project that will allow users to see a heat map representation of their mouse movements on the screen in real time. My goal is to make this API as dynamic as possible.
By dynamic, I mean I would like users to be able to use this API to generate the heat map in real time and then plug that heat map into their third party graphics software to view that heat map. (i.e Unity, React, Mobile, etc.. )
For testing purposes the third party graphics software that I am using is Unity.
I have created a .cs unity script that does the following per frame:
Start recording mouse locations.
Have ExternalProgram.exe generate a bitmap image using the
points that unity just recorded.
Then have Unity display the updated .bmp image on the screen.
Right now the problem that I am having is that the .bmp file is not being created when I use ProcessStartInfo in my unity script to run the .exe that is in charge of creating the .bmp image.
I've been debugging this code for the past week trying to figure out what is wrong with it. I know for a fact that Unity is successfully recording the mouse's location and passing those values to the .exe after calling ProcessStartInfo.
But for some reason it doesn't actually create the .bmp file. This is weird because if I independently run the ExternalProject in Visual Studio then it works just fine and it creates the .bmp file and shows the correct heat map representation on the image.
I figured that maybe starting a program and passing it tons of data and having that program create a file would be a lot of work for unity to do every single frame. (I am open to suggestions on ways to get around having to do that)
So I decided to just have the script record points for the first 15 seconds and then try to write that data to the .bmp file but that didn't work either.
Main program file for ExternalProject.exe
class Program
{
public static void Main(string[] args)
{
string arguments = "";
foreach (string arg in args)
{
arguments += arg;
}
Console.WriteLine("My Args: " + arguments + "--EOF");
bool noArguments = String.IsNullOrEmpty(arguments);
if (noArguments)
{
// Test input data
arguments = "(111,222)|(333,444)|(555,777)|(888,999)|(1000,1000)|(1000,1000)|(1000,1000)";
}
// The method ConvertStringToSignalsList() has already been tested and it works.
List<MouseMovementSignal> signals = ConvertStringToSignalsList(arguments);
CreateMouseHeatmap(signals);
Console.WriteLine("Press Enter to close...");
Console.ReadLine();
}
private static void CreateMouseHeatmap(List<MouseMovementSignal> argumentSignals)
{
int Height = 2100;
int Width = 3800;
List<MouseMovementSignal> mouseSignals
= argumentSignals; // Program perameters
//= getRecordedMouseData(); // DB
//= createFakeMouseSignals(Height, Width); // Generates fake signals
try
{
HeatmapStructure<MouseMovementSignal> mapper =
new HeatmapStructure<MouseMovementSignal>(Height, Width);
mapper.LoadSignalsFromObjects(mouseSignals);
// .BMP Image is created inside of this method !!
mapper.IdentifyHeatRegions();
//MouseMovementSignal mms = argumentSignals[argumentSignals.Count - 1];
//Console.WriteLine("Last: " + mms.Channels[0].Name + ": " + mms.Channels[0].Values[0] + ", "
// + mms.Channels[1].Name + ": " + mms.Channels[1].Values[0]);
//finalHeatmap.Save("MyFirstBitmap.bmp", System.Drawing.Imaging.ImageFormat.Bmp);
Console.WriteLine("Image actually Complete!!!!!!!!!!!!!!");
}
catch (Exception e)
{
Console.WriteLine("Error: " + e);
}
}
}
My Unity Script
public class HeatmapScript : MonoBehaviour {
private List<string> points;
private Camera cam;
private RawImage ri;
private string heatmapBmpPath = #"C:\Users\not-showing-you-my-file-structure-LOL\MyFirstBitmap.bmp";
private int frameCount = 0;
string pointsAsString = "";
Stopwatch sw;
// Use this for initialization
void Start() {
// Initialize the raw image.
cam = Camera.main;
GameObject imageObject = GameObject.FindGameObjectWithTag("imageView");
ri = imageObject.GetComponent<RawImage>();
// Initialize points list.
points = new List<string>();
sw = new Stopwatch();
sw.Start();
}
bool stop = false;
// Update is called once per frame.
void Update() {
float xValue = Input.mousePosition.x;
float yValue = Input.mousePosition.y;
Vector2 newPoint = new Vector2(xValue, yValue);
points.Add(newPoint.ToString());
int tSecs = 15000;// 15 seconds
// After 15 seconds of recording points pass them to the program that creates the heat map.
if (stop == false && sw.ElapsedMilliseconds > tSecs)
{
StartCoroutine("UpdateBMP");
UnityEngine.Debug.Log(points.Count);
stop = true;
}
//Update the raw image on the screen.
UnityEngine.Texture2D newTexture = CreateTextureFromBitmap(heatmapBmpPath);
//Set the RawImage to the size of the scren.
ri.texture = newTexture;
ri.rectTransform.sizeDelta = new Vector2(Screen.width, Screen.height);
frameCount++;
}
IEnumerator UpdateBMP()
{
// Show mouse position in unity environment
float xValue = Input.mousePosition.x;
float yValue = Input.mousePosition.y;
Vector2 newPoint = new Vector2(xValue, yValue);
points.Add(newPoint.ToString());
// EX:
// (123,123)|(123,123)|(123,123)|(123,123)|(123,123)|(123,123)
// display list contents without loop
pointsAsString = string.Join("|", points.ToArray());
// Every frame call Behavior's Program.cs that calls HeatmapStructure.cs to update .bmp file
ProcessStartInfo processInfo = new ProcessStartInfo();
processInfo.FileName = #"C:\Users\not-showing-you-my-file-structure-LOL\ExternalProgram.exe";
processInfo.UseShellExecute = false;
processInfo.Arguments = pointsAsString;
Process process = Process.Start(processInfo);
yield return null;
}
private UnityEngine.Texture2D CreateTextureFromBitmap(string completeFilePath)
{
BMPLoader loader = new BMPLoader();
BMPImage img = loader.LoadBMP(completeFilePath);
UnityEngine.Texture2D myTexture = img.ToTexture2D();
UnityEngine.Debug.Log("File Size: " + img.header.filesize);
return myTexture;
}
}
HeatmapStructure.cs class
public class HeatmapStructure<T> where T : ISignal
{
public class COGPoint
{
public double X, Y, Z;
//public Color Color;
public byte Intensity;
public bool isD3Point = false; // 3D Point check
public const double DEFAULT_AXIS_LOC = 0.0001;
public COGPoint()
{
//Color = Color.Blue;
Intensity = 0;
}
// NOTE: double z has a default value therefore it is optional
public COGPoint(byte intensity, double x, double y, double z = DEFAULT_AXIS_LOC)
{
this.X = x;
this.Y = y;
this.Z = z; // Optional
//Color = Color.Blue; // Cold: Blue / Hot: Red
this.Intensity = intensity;
if (z != DEFAULT_AXIS_LOC)
{
isD3Point = true;
}
}
public override string ToString()
{
string output = (isD3Point == true) ?
("(x,y,z) " + X + "," + Y + "," + Z) : ("(x,y) " + X + "," + Y); // 3D : 2D
output += //" Color: " + Color.ToString() +
" Intensity: " + Intensity;
return output;
}
}
private List<COGPoint> points;
private int Height;
private int Width;
public HeatmapStructure(int Height, int Width)
{
this.Height = Height;
this.Width = Width;
points = new List<COGPoint>();
}
private Bitmap CreateIntensityMask(Bitmap bSurface, List<COGPoint> aHeatPoints)
{
// Create new graphics surface from memory bitmap
Graphics DrawSurface = Graphics.FromImage(bSurface);
// Set background color to white so that pixels can be correctly colorized
DrawSurface.Clear(Color.White);
// Traverse heat point data and draw masks for each heat point
foreach (COGPoint DataPoint in aHeatPoints)
{
// Render current heat point on draw surface
DrawHeatPoint(DrawSurface, DataPoint, 45);
}
return bSurface;
}
// TODO: How to draw updating Bitmap in unity in real time ??
private void DrawHeatPoint(Graphics Canvas, COGPoint HeatPoint, int Radius)
{
// Create points generic list of points to hold circumference points
List<Point> CircumferencePointsList = new List<Point>();
// Create an empty point to predefine the point struct used in the circumference loop
Point CircumferencePoint;
// Create an empty array that will be populated with points from the generic list
Point[] CircumferencePointsArray;
// Calculate ratio to scale byte intensity range from 0-255 to 0-1
float fRatio = 1F / Byte.MaxValue;
// Precalulate half of byte max value
byte bHalf = Byte.MaxValue / 2;
// Flip intensity on it's center value from low-high to high-low
int iIntensity = (byte)(HeatPoint.Intensity - ((HeatPoint.Intensity - bHalf) * 2));
// Store scaled and flipped intensity value for use with gradient center location
float fIntensity = iIntensity * fRatio;
// Loop through all angles of a circle
// Define loop variable as a double to prevent casting in each iteration
// Iterate through loop on 10 degree deltas, this can change to improve performance
for (double i = 0; i <= 360; i += 10)
{
// Replace last iteration point with new empty point struct
CircumferencePoint = new Point();
// Plot new point on the circumference of a circle of the defined radius
// Using the point coordinates, radius, and angle
// Calculate the position of this iterations point on the circle
CircumferencePoint.X = Convert.ToInt32(HeatPoint.X + Radius * Math.Cos(ConvertDegreesToRadians(i)));
CircumferencePoint.Y = Convert.ToInt32(HeatPoint.Y + Radius * Math.Sin(ConvertDegreesToRadians(i)));
// Add newly plotted circumference point to generic point list
CircumferencePointsList.Add(CircumferencePoint);
}
// Populate empty points system array from generic points array list
// Do this to satisfy the datatype of the PathGradientBrush and FillPolygon methods
CircumferencePointsArray = CircumferencePointsList.ToArray();
// Create new PathGradientBrush to create a radial gradient using the circumference points
PathGradientBrush GradientShaper = new PathGradientBrush(CircumferencePointsArray);
// Create new color blend to tell the PathGradientBrush what colors to use and where to put them
ColorBlend GradientSpecifications = new ColorBlend(3);
// Define positions of gradient colors, use intesity to adjust the middle color to
// show more mask or less mask
GradientSpecifications.Positions = new float[3] { 0, fIntensity, 1 };
// Define gradient colors and their alpha values, adjust alpha of gradient colors to match intensity
GradientSpecifications.Colors = new Color[3]
{
Color.FromArgb(0, Color.White),
Color.FromArgb(HeatPoint.Intensity, Color.Black),
Color.FromArgb(HeatPoint.Intensity, Color.Black)
};
// Pass off color blend to PathGradientBrush to instruct it how to generate the gradient
GradientShaper.InterpolationColors = GradientSpecifications;
// Draw polygon (circle) using our point array and gradient brush
Canvas.FillPolygon(GradientShaper, CircumferencePointsArray);
}
private double ConvertDegreesToRadians(double degrees)
{
double radians = (Math.PI / 180) * degrees;
return (radians);
}
// old name : button1_Click
public Bitmap IdentifyHeatRegions()
{
// Create new memory bitmap the same size as the picture box
Bitmap bMap = new Bitmap(Width, Height);
// Call CreateIntensityMask, give it the memory bitmap, and use it's output to set the picture box image
Bitmap bm = CreateIntensityMask(bMap, points);
Bitmap coloredBitmap = Colorize(bm, 243); // <-- NOTE: should be 255. But my palette.bmp is 243x5
coloredBitmap.Save("MyFirstBitmap.bmp", System.Drawing.Imaging.ImageFormat.Bmp);
return coloredBitmap;
}
public static Bitmap Colorize(Bitmap Mask, byte Alpha)
{
// Create new bitmap to act as a work surface for the colorization process
Bitmap Output = new Bitmap(Mask.Width, Mask.Height, PixelFormat.Format32bppArgb);
// Create a graphics object from our memory bitmap so we can draw on it and clear it's drawing surface
Graphics Surface = Graphics.FromImage(Output);
Surface.Clear(Color.Transparent);
// Build an array of color mappings to remap our greyscale mask to full color
// Accept an alpha byte to specify the transparancy of the output image
ColorMap[] Colors = CreatePaletteIndex(Alpha);
// Create new image attributes class to handle the color remappings
// Inject our color map array to instruct the image attributes class how to do the colorization
ImageAttributes Remapper = new ImageAttributes();
try
{
Remapper.SetRemapTable(Colors);
}
catch (Exception e)
{
Console.WriteLine(e);
}
// Draw our mask onto our memory bitmap work surface using the new color mapping scheme
Surface.DrawImage(Mask, new Rectangle(0, 0, Mask.Width, Mask.Height), 0, 0, Mask.Width, Mask.Height, GraphicsUnit.Pixel, Remapper);
// Send back newly colorized memory bitmap
return Output;
}
private static ColorMap[] CreatePaletteIndex(byte Alpha)
{
ColorMap[] OutputMap = new ColorMap[Alpha + 1];
// Change this path to wherever you saved the palette image.
Bitmap Palette = (Bitmap)Bitmap.FromFile(#"C:\Users\cdowns\Desktop\palette.bmp");
// Loop through each pixel and create a new color mapping
try
{
for (int X = 0; X <= Alpha; X++)
{
OutputMap[X] = new ColorMap();
OutputMap[X].OldColor = Color.FromArgb(X, X, X);
OutputMap[X].NewColor = Color.FromArgb(Alpha, Palette.GetPixel(X, 0));
}
}
catch (Exception e) {
Console.WriteLine(e);
}
return OutputMap;
}
public void LoadSignalsFromObjects(List<T> allSignals) // ISignal Object
{
Random random = new Random();
foreach (T signal in allSignals)
{
COGPoint newPoint = new COGPoint();
if (allSignals[0].GetType() == typeof(MouseMovementSignal))
{
string axis1 = signal.Channels[0].Name;
List<double> value1 = signal.Channels[0].Values;
string axis2 = signal.Channels[1].Name;
List<double> value2 = signal.Channels[1].Values;
// Make sure to enter signals into database in this format
//Console.WriteLine(axis1 + " " + axis2);
newPoint.X = value1[0];
newPoint.Y = value2[0];
// TOOD: Implement
newPoint.Intensity = (byte)random.Next(0, 120);
// Display newPoint values
//Console.WriteLine("COGPoint Numbers: X: " + newPoint.X + " , Y: " + newPoint.Y
// + /*" Color: " + newPoint.Color + */" Intensity: " + newPoint.Intensity);
}
else if (allSignals[0].GetType() == typeof(EyePosition))
{
}
// Add to main list of heat points
points.Add(newPoint);
}
}
}
Expected result is that the .bmp image is created after the first 15 seconds.
(P.S. I am very new to both Unity and C# to I am probably doing this completely wrong. I am open to an entirely new idea for going about making this work. Thanks)
After doing a little more browsing on here for similar problems that other people have had I found this post pictureBox.Image.Save() in c# doesn't work. It answered my question regarding why my .bmp wasn't being generated.
It turns out that my program was working correctly after all. It was correctly generating the .bmp file. However, when I called ProcessStartInfo from within unity to run the ExternalProgram.exe that called Bitmap.save("filename.bmp") the working directory changed. Therefore, the image was not being saved in the location that I was expecting to find it in.
I have several PDF files, using a Windows application (C#), I need to find out whether the PDF files has overlapping text or not. How can I do it, is there any free third party DLLs to achieve this?
All I have got now is third party DLLs which can get the text/images from a PDF.
My PDFs are full of texts and images. Here, one line of text is printed on top of another line or few texts are printed on top of some images. These kind of overlapping needs to found.
As you can see in the image, those overlapping might have occurred because of bounding boxes overlap and as well as glyphs contours overlap. So these two occurrences in the PDF needs to be found. My PDF doesn't contain any annotations. So overlapping occurs only in the content of pdf. We don't use poor-man's-bold technique for fatter glyph and if that occurs then it shoul be consider as overlapping.
There is not going to be any transparent images in the PDF, only image we might have is the logo or the digital signature at the bottom of the page, any text overlaps this should be considered as overlapping.
PDFs are not created from image(scan). From some text editor it has been created.
The OP clarified in comments:
those overlapping might have occurred because of bounding boxes overlap and as well as glyphs contours overlap. So these two occurrences in the PDF needs to be found.
Whenever the glyph contours themselves overlap, their bounding boxes also overlap.
Thus, it suffices to check for overlapping bounding boxes.
only image we might have is the logo or the digital signature at the bottom of the page, any text overlaps this should be considered as overlapping.
Thus, for text overlapping images we do not need to check whether a blank area in the image is overlapped.
My PDF files doesnt have any annotations.
Thus, we only need to check the page contents (including contents of form xobjects referenced from the page content, allowing recursion).
Furthermore the OP only mentioned text and images. Thus, we can ignore vector graphics.
An approach using iText 7
As I'm more into Java, I first created a prove-of-concept in Java and ported it to .Net later.
Both for Java and .Net the line of action is the same:
We create a event listener for the iText 7 parsing framework which (while processing a page) collects the bounding boxes of text and image elements and eventually can be asked to check whether there are any occurrences of text overlapping text or image.
We parse the content of the page in question using an instance of that event listener class and query it for overlaps. If more pages are to be checked, this can be done over and over again with a new event listener instance for each page.
iText 7 for .Net
The event listener might look like this:
class OverlappingTextSearchingStrategy : IEventListener
{
static List<Vector> UNIT_SQUARE_CORNERS = new List<Vector> { new Vector(0, 0, 1), new Vector(1, 0, 1), new Vector(1, 1, 1), new Vector(0, 1, 1) };
ICollection<Rectangle> imageRectangles = new HashSet<Rectangle>();
ICollection<Rectangle> textRectangles = new HashSet<Rectangle>();
public void EventOccurred(IEventData data, EventType type)
{
if (data is ImageRenderInfo) {
ImageRenderInfo imageData = (ImageRenderInfo)data;
Matrix ctm = imageData.GetImageCtm();
List<Rectangle> cornerRectangles = new List<Rectangle>(UNIT_SQUARE_CORNERS.Count);
foreach (Vector unitCorner in UNIT_SQUARE_CORNERS)
{
Vector corner = unitCorner.Cross(ctm);
cornerRectangles.Add(new Rectangle(corner.Get(Vector.I1), corner.Get(Vector.I2), 0, 0));
}
Rectangle boundingBox = Rectangle.GetCommonRectangle(cornerRectangles.ToArray());
Console.WriteLine("Adding image bounding rectangle {0}.", boundingBox);
imageRectangles.Add(boundingBox);
} else if (data is TextRenderInfo) {
TextRenderInfo textData = (TextRenderInfo)data;
Rectangle ascentRectangle = textData.GetAscentLine().GetBoundingRectangle();
Rectangle descentRectangle = textData.GetDescentLine().GetBoundingRectangle();
Rectangle boundingBox = Rectangle.GetCommonRectangle(ascentRectangle, descentRectangle);
if (boundingBox.GetHeight() == 0 || boundingBox.GetWidth() == 0)
Console.WriteLine("Ignoring empty text bounding rectangle {0} for \"{1}\".", boundingBox, textData.GetText());
else
{
Console.WriteLine("Adding text bounding rectangle {0} for \"{1}\" with 0.5 margins.", boundingBox, textData.GetText());
textRectangles.Add(boundingBox.ApplyMargins<Rectangle>(0.5f, 0.5f, 0.5f, 0.5f, false));
}
} else if (data is PathRenderInfo) {
// TODO
} else if (data != null)
{
Console.WriteLine("Ignored {0} event, class {1}.", type, data.GetType().Name);
}
else
{
Console.WriteLine("Ignored {0} event with null data.", type);
}
}
public ICollection<EventType> GetSupportedEvents()
{
// Support all events
return null;
}
public bool foundOverlappingText()
{
bool result = false;
List<Rectangle> textRectangleList = new List<Rectangle>(textRectangles);
while (textRectangleList.Count > 0)
{
Rectangle testRectangle = textRectangleList[textRectangleList.Count - 1];
textRectangleList.RemoveAt(textRectangleList.Count - 1);
foreach (Rectangle rectangle in textRectangleList)
{
if (intersect(testRectangle, rectangle))
{
Console.WriteLine("Found text intersecting text with bounding boxes {0} at {1},{2} and {3} at {4},{5}.",
testRectangle, testRectangle.GetX(), testRectangle.GetY(), rectangle, rectangle.GetX(), rectangle.GetY());
result = true;// if only the fact counts, do instead: return true
}
}
foreach (Rectangle rectangle in imageRectangles)
{
if (intersect(testRectangle, rectangle))
{
Console.WriteLine("Found text intersecting image with bounding boxes {0} at {1},{2} and {3} at {4},{5}.",
testRectangle, testRectangle.GetX(), testRectangle.GetY(), rectangle, rectangle.GetX(), rectangle.GetY());
result = true;// if only the fact counts, do instead: return true
}
}
}
return result;
}
bool intersect(Rectangle a, Rectangle b)
{
return intersect(a.GetLeft(), a.GetRight(), b.GetLeft(), b.GetRight()) &&
intersect(a.GetBottom(), a.GetTop(), b.GetBottom(), b.GetTop());
}
bool intersect(float start1, float end1, float start2, float end2)
{
if (start1 < start2)
return start2 <= end1;
else
return start1 <= end2;
}
}
This event listener can be used like this:
PdfReader reader = new PdfReader(pdf);
PdfDocument document = new PdfDocument(reader);
PdfDocumentContentParser contentParser = new PdfDocumentContentParser(document);
OverlappingTextSearchingStrategy strategy = contentParser.ProcessContent(page, new OverlappingTextSearchingStrategy());
bool foundOverlaps = strategy.foundOverlappingText();
iText 7 for Java
The event listener might look like this:
public class OverlappingTextSearchingStrategy implements IEventListener {
static List<Vector> UNIT_SQUARE_CORNERS = Arrays.asList(new Vector(0,0,1), new Vector(1,0,1), new Vector(1,1,1), new Vector(0,1,1));
Set<Rectangle> imageRectangles = new HashSet<>();
Set<Rectangle> textRectangles = new HashSet<>();
#Override
public void eventOccurred(IEventData data, EventType type) {
if (data instanceof ImageRenderInfo) {
ImageRenderInfo imageData = (ImageRenderInfo) data;
Matrix ctm = imageData.getImageCtm();
List<Rectangle> cornerRectangles = new ArrayList<>(UNIT_SQUARE_CORNERS.size());
for (Vector unitCorner : UNIT_SQUARE_CORNERS) {
Vector corner = unitCorner.cross(ctm);
cornerRectangles.add(new Rectangle(corner.get(Vector.I1), corner.get(Vector.I2), 0, 0));
}
Rectangle boundingBox = Rectangle.getCommonRectangle(cornerRectangles.toArray(new Rectangle[cornerRectangles.size()]));
logger.info(String.format("Adding image bounding rectangle %s.", boundingBox));
imageRectangles.add(boundingBox);
} else if (data instanceof TextRenderInfo) {
TextRenderInfo textData = (TextRenderInfo) data;
Rectangle ascentRectangle = textData.getAscentLine().getBoundingRectangle();
Rectangle descentRectangle = textData.getDescentLine().getBoundingRectangle();
Rectangle boundingBox = Rectangle.getCommonRectangle(ascentRectangle, descentRectangle);
if (boundingBox.getHeight() == 0 || boundingBox.getWidth() == 0)
logger.info(String.format("Ignoring empty text bounding rectangle %s for '%s'.", boundingBox, textData.getText()));
else {
logger.info(String.format("Adding text bounding rectangle %s for '%s' with 0.5 margins.", boundingBox, textData.getText()));
textRectangles.add(boundingBox.applyMargins(0.5f, 0.5f, 0.5f, 0.5f, false));
}
} else if (data instanceof PathRenderInfo) {
// TODO: vector graphics
} else if (data != null) {
logger.fine(String.format("Ignored %s event, class %s.", type, data.getClass().getSimpleName()));
} else {
logger.fine(String.format("Ignored %s event with null data.", type));
}
}
#Override
public Set<EventType> getSupportedEvents() {
// Support all events
return null;
}
public boolean foundOverlappingText() {
boolean result = false;
List<Rectangle> textRectangleList = new ArrayList<>(textRectangles);
while (!textRectangleList.isEmpty())
{
Rectangle testRectangle = textRectangleList.remove(textRectangleList.size() - 1);
for (Rectangle rectangle : textRectangleList) {
if (intersect(testRectangle, rectangle)) {
logger.info(String.format("Found text intersecting text with bounding boxes %s at %s,%s and %s at %s,%s.",
testRectangle, testRectangle.getX(), testRectangle.getY(), rectangle, rectangle.getX(), rectangle.getY()));
result = true;// if only the fact counts, do instead: return true
}
}
for (Rectangle rectangle : imageRectangles) {
if (intersect(testRectangle, rectangle)) {
logger.info(String.format("Found text intersecting image with bounding boxes %s at %s,%s and %s at %s,%s.",
testRectangle, testRectangle.getX(), testRectangle.getY(), rectangle, rectangle.getX(), rectangle.getY()));
result = true;// if only the fact counts, do instead: return true
}
}
}
return result;
}
boolean intersect(Rectangle a, Rectangle b) {
return intersect(a.getLeft(), a.getRight(), b.getLeft(), b.getRight()) &&
intersect(a.getBottom(), a.getTop(), b.getBottom(), b.getTop());
}
boolean intersect(float start1, float end1, float start2, float end2) {
if (start1 < start2)
return start2 <= end1;
else
return start1 <= end2;
}
Logger logger = Logger.getLogger(OverlappingTextSearchingStrategy.class.getName());
}
This event listener can be used like this:
PdfReader reader = new PdfReader(pdf);
PdfDocument document = new PdfDocument(reader);
PdfDocumentContentParser contentParser = new PdfDocumentContentParser(document);
OverlappingTextSearchingStrategy strategy = contentParser.processContent(pageNumber, new OverlappingTextSearchingStrategy());
boolean foundOverlaps = strategy.foundOverlappingText();
Remarks
As you can see I don't store the text bounding boxes as they are but instead
boundingBox.applyMargins(0.5f, 0.5f, 0.5f, 0.5f, false),
i.e. slightly smaller boxes. This is done to prevent false positives which otherwise might occur for tightly set text or text with kerning applied. You may have to fine tune the margin values here.
It may be as easy as the example above or you have to implement your own reader for this.
If you have not the full control over your PDF files, you have no chance to solve your problem. The defined boxes can be transformed later on. So you have to parse the whole file, too keep track of the box position and form. Additionally some boxes may be on top of other boxes, but render without any collision on the pixel level.
Than you will run into the next problem. Each PDF implementation has different errors. So your system may render the text perfectly but not the printer of your customer.
Welcome to hell ;)
Each support guy will tell you that they obey the standard. The others must have implemented their PDF library faulty. Because your customers data will be confident, you cannot proof them wrong. You may find some errors with your test data, but never ever the same errors of your customer documents.
Run and hide as long as you have not become the PDF expert of your company.
Here is a dirty "general" method: render your text without the text in bitmap. render the page with your text in another bitmap, compare the area with your text. But this will need a monochrome background. But the load will be really high. But this document looks like a form. Create a form and fill out the form boxes. So you will have no problems and you will even get correct results, fills the form with another program
Hello I have a code sample that uses not free library, but I think other libraries should have similar functionality, so you may use it as the idea:
Before use the following code sample please ensure that you use the latest version of the Apitron PDF Kit.
using System;
using System.Collections.Generic;
using System.IO;
using Apitron.PDF.Kit.FixedLayout;
using Apitron.PDF.Kit.FixedLayout.Content;
using Apitron.PDF.Kit.FixedLayout.PageProperties;
using FixedLayout.Resources;
using FixedLayout.ContentElements;
/// <summary>
/// Gets all text boundaries.
/// </summary>
/// <param name="elements">The elements.</param>
/// <param name="boundaries">The boundaries.</param>
public void GetAllTextBoundaries(IContentElementsEnumerator elements, IList<Boundary> boundaries, Boundary offset)
{
// We dont count drawings and images here - only text;
if(elements == null)
{
return;
}
foreach (IContentElement element in elements)
{
TextContentElement text = element as TextContentElement;
if (text != null)
{
foreach (TextSegment segment in text.Segments)
{
Boundary currentBoundary = segment.Boundary;
if (offset != null)
{
currentBoundary = new Boundary(currentBoundary.Left + offset.Left, currentBoundary.Bottom + offset.Bottom, currentBoundary.Right + offset.Left, currentBoundary.Top + offset.Bottom);
}
boundaries.Add(currentBoundary);
}
}
else if (element is FormContentElement)
{
Boundary currentBoundary = (element as FormContentElement).Boundary;
if (offset != null)
{
currentBoundary = new Boundary(currentBoundary.Left + offset.Left, currentBoundary.Bottom + offset.Bottom, currentBoundary.Right + offset.Left, currentBoundary.Top + offset.Bottom);
}
this.GetAllTextBoundaries((element as FormContentElement).FormXObject.Elements, boundaries, currentBoundary);
}
}
}
/// <summary>
/// Checks if text is overlapped.
/// </summary>
/// <returns></returns>
public bool CheckIfTextIsOverlapped(string fileName)
{
const double overlapMax = 5;
using (System.IO.Stream stream = new FileStream(fileName, FileMode.Open, FileAccess.ReadWrite))
{
using (FixedDocument document = new FixedDocument(stream))
{
foreach (Page page in document.Pages)
{
IList<Boundary> boundaries = new List<Boundary>();
foreach (Annotation annotation in page.Annotations)
{
// Actually we need only Normal state, but will check all - to be sure.
if(annotation.Appearance.Normal != null)
{
this.GetAllTextBoundaries(annotation.Appearance.Normal.Elements, boundaries, annotation.Boundary);
}
}
IContentElementsEnumerator elements = page.Elements;
this.GetAllTextBoundaries(elements, boundaries, null);
for (int i = 0; i < boundaries.Count; i++)
{
for (int j = i + 1; j < boundaries.Count; j++)
{
Boundary b1 = boundaries[i];
Boundary b2 = boundaries[j];
double x1 = Math.Max(b1.Left, b2.Left);
double y1 = Math.Max(b1.Bottom, b2.Bottom);
double x2 = Math.Min(b1.Right, b2.Right);
double y2 = Math.Min(b1.Top, b2.Top);
// So we have intersection
if (x1 < x2 && y1 < y2)
{
if (x1 - x2 >= overlapMax || y1 - y2 >= overlapMax)
{
return true;
}
}
}
}
}
}
}
return false;
}
I have a Kinect WPF Application that takes images from the Kinect, does some feature detection using EmguCV (A C# opencv wrapper) and displays the output on the using a WPF image.
I have had this working before, but the application now refuses to update the screen image when the imagesource is written to, but I have not changed the way it works.
the Image(called video) is written to as such:
video.Source = bitmapsource;
in the colorframeready event handler.
This works fine until I introduce some opencv code before the imagesource is written to. It does not matter what source is used, so I don't think it is a conflict there. I have narrowed down the offending EmguCV code to this line:
RecentKeyPoints = surfCPU.DetectKeyPointsRaw(ImageRecent, null);
which jumps straight into the opencv code. It is worth noting that:
ImageRecent has completely different origins to the bitmapsource updating the screen.
Reading video.Source returns the bitmapsource, so it seems to be writing correctly, just not updating the screen.
Let me know if you want any more information...
void nui_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
// Checks for a recent Depth Image
if (!TrackingReady) return;
// Stores image
using (ColorImageFrame colorImageFrame = e.OpenColorImageFrame())
{
if (colorImageFrame != null)
{
if (FeatureTracker.ColourImageRecent == null)
//allocate the first time
FeatureTracker.ColourImageRecent = new byte[colorImageFrame.PixelDataLength];
colorImageFrame.CopyPixelDataTo(FeatureTracker.ColourImageRecent);
}
else return;
}
FeatureTracker.FeatureDetect(nui);
//video.Source = FeatureTracker.ColourImageRecent.ToBitmapSource();
video.Source = ((Bitmap)Bitmap.FromFile("test1.png")).ToBitmapSource();
TrackingReady = false;
}
public Bitmap FeatureDetect(KinectSensor nui)
{
byte[] ColourClone = new byte[ColourImageRecent.Length];
Array.Copy(ColourImageRecent, ColourClone, ColourImageRecent.Length);
Bitmap test = (Bitmap)Bitmap.FromFile("test1.png");
test.RotateFlip(RotateFlipType.RotateNoneFlipY);
Image<Gray, Byte> ImageRecent = new Image<Gray, byte>(test);
SURFDetector surfCPU = new SURFDetector(2000, false);
VectorOfKeyPoint RecentKeyPoints;
Matrix<int> indices;
Matrix<float> dist;
Matrix<byte> mask;
bool MatchFailed = false;
// extract SURF features from the object image
RecentKeyPoints = surfCPU.DetectKeyPointsRaw(ImageRecent, null);
//Matrix<float> RecentDescriptors = surfCPU.ComputeDescriptorsRaw(ImageRecent, null, RecentKeyPoints);
//MKeyPoint[] RecentPoints = RecentKeyPoints.ToArray();
// don't feature detect on first attempt, just store image details for next attempt
#region
/*
if (KeyPointsOld == null)
{
KeyPointsOld = RecentKeyPoints;
PointsOld = RecentPoints;
DescriptorsOld = RecentDescriptors;
return ImageRecent.ToBitmap();
}
*/
#endregion
// Attempt to match points to their nearest neighbour
#region
/*
BruteForceMatcher SURFmatcher = new BruteForceMatcher(BruteForceMatcher.DistanceType.L2F32);
SURFmatcher.Add(RecentDescriptors);
int k = 5;
indices = new Matrix<int>(DescriptorsOld.Rows, k);
dist = new Matrix<float>(DescriptorsOld.Rows, k);
*/
// Match features, provide the top k matches
//SURFmatcher.KnnMatch(DescriptorsOld, indices, dist, k, null);
// Create mask and set to allow all features
//mask = new Matrix<byte>(dist.Rows, 1);
//mask.SetValue(255);
#endregion
//Features2DTracker.VoteForUniqueness(dist, 0.8, mask);
// Check number of good maches and for error and end matching if true
#region
//int nonZeroCount = CvInvoke.cvCountNonZero(mask);
//if (nonZeroCount < 5) MatchFailed = true;
/*
try
{
nonZeroCount = Features2DTracker.VoteForSizeAndOrientation(RecentKeyPoints, KeyPointsOld, indices, mask, 1.5, 20);
}
catch (SystemException)
{
MatchFailed = true;
}
if (nonZeroCount < 5) MatchFailed = true;
if (MatchFailed)
{
return ImageRecent.ToBitmap();
}
*/
#endregion
//DepthMapColourCoordsRecent = CreateDepthMap(nui, DepthImageRecent);
//PointDist[] FeatureDistances = DistanceToFeature(indices, mask, RecentPoints);
//Image<Rgb,Byte> rgbimage = ImageRecent.Convert<Rgb, Byte>();
//rgbimage = DrawPoints(FeatureDistances, rgbimage);
// Store recent image data for next feature detect.
//KeyPointsOld = RecentKeyPoints;
//PointsOld = RecentPoints;
//DescriptorsOld = RecentDescriptors;
//CreateDepthMap(nui, iva);
//rgbimage = CreateDepthImage(DepthMapColourCoordsRecent, rgbimage);
// Convert image back to a bitmap
count++;
//Bitmap bitmap3 = rgbimage.ToBitmap();
//bitmapstore = bitmap3;
//bitmap3.Save("test" + count.ToString() + ".png");
return null;
}
This is a little late, but I had a similar problem and thought I'd share my solution.
In my case I was processing the depth stream. The default resolution was 640x480, and Emgu just wasn't able to process the image fast enough to keep up with the frameready handler. As soon as I reduced the depth stream resolution to 320x240 the problem went away.
I also went a bit further and moved my image processing to a different thread which sped it up even more (do a search for ComponentDispatcher.ThreadIdle). I'm still not able to do 640x480 at a reasonable frame rate, but at least the image renders so I can see what's going on.
I have a UISlider. It is used to navigate quickly through a PDF. Whenever the threshold for the next page is reached, I display a UIView next to the slider's knob that contains a small preview of the target page.
The slider code looks is below this (some parts stripped). If the next page is reached, a new preview is generated, Otherwise, the existing one is moved along the slider.
I get various effects:
If previewing many many pages, the app crashes at
MonoTouch.CoreGraphics.CGContext.Dispose (bool) <0x00047>
Oct 11 17:21:13 unknown UIKitApplication:com.brainloop.brainloopbrowser[0x1a2d][2951] : at MonoTouch.CoreGraphics.CGContext.Finalize () <0x0002f>
or if I remove the calls to Dispose() in the last method: [NSAutoreleasePool release]: This pool has already been released, do not drain it (double release).
From looking at the code, does somebody have an idea what's wrong? Or is the whole approach of using a thread wrong?
this.oScrollSlider = new UISlider ();
this.oScrollSlider.TouchDragInside += delegate( object oSender, EventArgs oArgs )
{
this.iCurrentPage = (int)Math.Round (oScrollSlider.Value);
if (this.iCurrentPage != this.iLastScrollSliderPage)
{
this.iLastScrollSliderPage = this.iCurrentPage;
this.RenderScrollPreviewImage(this.iCurrentPage);
}
};
this.oScrollSlider.ValueChanged += delegate
{
if (this.oScrollSliderPreview != null)
{
this.oScrollSliderPreview.RemoveFromSuperview ();
this.oScrollSliderPreview.Dispose();
this.oScrollSliderPreview = null;
}
// Go to the selected page.
};
The method creating the preview is spinning off a new thread. If the user changes pages whil the thread is still going, it is aborted and the next page is previewed:
private void RenderScrollPreviewImage (int iPage)
{
// Create a new preview view if not there.
if(this.oScrollSliderPreview == null)
{
SizeF oSize = new SizeF(150, 200);
RectangleF oFrame = new RectangleF(new PointF (this.View.Bounds.Width - oSize.Width - 50, this.GetScrollSliderOffset (oSize)), oSize);
this.oScrollSliderPreview = new UIView(oFrame);
this.oScrollSliderPreview.BackgroundColor = UIColor.White;
this.View.AddSubview(this.oScrollSliderPreview);
UIActivityIndicatorView oIndicator = new UIActivityIndicatorView(UIActivityIndicatorViewStyle.Gray);
oIndicator.Center = new PointF(this.oScrollSliderPreview.Bounds.Width/2, this.oScrollSliderPreview.Bounds.Height/2);
this.oScrollSliderPreview.AddSubview(oIndicator);
oIndicator.StartAnimating();
}
// Remove all subviews, except the activity indicator.
if(this.oScrollSliderPreview.Subviews.Length > 0)
{
foreach(UIView oSubview in this.oScrollSliderPreview.Subviews)
{
if(!(oSubview is UIActivityIndicatorView))
{
oSubview.RemoveFromSuperview();
}
}
}
// Kill the currently running thread that renders a preview.
if(this.oRenderScrollPreviewImagesThread != null)
{
try
{
this.oRenderScrollPreviewImagesThread.Abort();
}
catch(ThreadAbortException)
{
// Expected.
}
}
// Start a new rendering thread.
this.oRenderScrollPreviewImagesThread = new Thread (delegate()
{
using (var oPool = new NSAutoreleasePool())
{
try
{
// Create a quick preview.
UIImageView oImgView = PdfViewerHelpers.GetLowResPagePreview (this.oPdfDoc.GetPage (iPage), new RectangleF (0, 0, 150, 200));
this.InvokeOnMainThread(delegate
{
if(this.oScrollSliderPreview != null)
{
oImgView.Center = new PointF(this.oScrollSliderPreview.Bounds.Width/2, this.oScrollSliderPreview.Bounds.Height/2);
// Add the PDF image to the preview view.
this.oScrollSliderPreview.AddSubview(oImgView);
}
});
}
catch (Exception)
{
}
}
});
// Start the thread.
this.oRenderScrollPreviewImagesThread.Start ();
}
To render the PDF image, I use this:
internal static UIImageView GetLowResPagePreview (CGPDFPage oPdfPage, RectangleF oTargetRect)
{
RectangleF oPdfPageRect = oPdfPage.GetBoxRect (CGPDFBox.Media);
// If preview is requested for the PDF index view, render a smaller version.
float fAspectScale = 1.0f;
if (!oTargetRect.IsEmpty)
{
fAspectScale = GetAspectZoomFactor (oTargetRect.Size, oPdfPageRect.Size, false);
// Resize the PDF page so that it fits the target rectangle.
oPdfPageRect = new RectangleF (new PointF (0, 0), GetFittingBox (oTargetRect.Size, oPdfPageRect.Size));
}
// Create a low res image representation of the PDF page to display before the TiledPDFView
// renders its content.
int iWidth = Convert.ToInt32 ( oPdfPageRect.Size.Width );
int iHeight = Convert.ToInt32 ( oPdfPageRect.Size.Height );
CGColorSpace oColorSpace = CGColorSpace.CreateDeviceRGB();
CGBitmapContext oContext = new CGBitmapContext(null, iWidth, iHeight, 8, iWidth * 4, oColorSpace, CGImageAlphaInfo.PremultipliedLast);
// First fill the background with white.
oContext.SetFillColor (1.0f, 1.0f, 1.0f, 1.0f);
oContext.FillRect (oPdfPageRect);
// Scale the context so that the PDF page is rendered
// at the correct size for the zoom level.
oContext.ScaleCTM (fAspectScale, fAspectScale);
oContext.DrawPDFPage (oPdfPage);
CGImage oImage = oContext.ToImage();
UIImage oBackgroundImage = UIImage.FromImage( oImage);
oContext.Dispose();
oImage.Dispose ();
oColorSpace.Dispose ();
UIImageView oBackgroundImageView = new UIImageView (oBackgroundImage);
oBackgroundImageView.Frame = new RectangleF (new PointF (0, 0), oPdfPageRect.Size);
oBackgroundImageView.ContentMode = UIViewContentMode.ScaleToFill;
oBackgroundImageView.UserInteractionEnabled = false;
oBackgroundImageView.AutoresizingMask = UIViewAutoresizing.None;
return oBackgroundImageView;
}
Avoid Thread.Abort().
Yeah, here are some links talking about it:
http://www.interact-sw.co.uk/iangblog/2004/11/12/cancellation
http://haacked.com/archive/2004/11/12/how-to-stop-a-thread.aspx
If you can use the .Net 4.0 features, go with them instead. Using a Task<T> is probably easier to work with in your case.
Also, I think it would be helpful to create some throttling and only start your new thread after 25-100 milliseconds of inactivity from the user.
Get rid of Thread.Abort() as Jonathan suggests.
Don't start a new thread for each image and instead have one background thread with a work queue. Then simply put the most important page to the front of the queue so it renders as soon as possible. You can also optionally limit the size of the queue or remove unneeded items from it.