I'm experimenting a bit with textual comparison/basic plagiarism detection, and want to try this on a website-to-website basis. However, I'm a bit stuck in finding a proper way to process the text.
How would you process and compare the content of two websites for plagiarism?
I'm thinking something like this pseudo-code:
// extract text
foreach website in websites
crawl website - store structure so pages are only scanned once
extract text blocks from all pages - store this is in list
// compare
foreach text in website1.textlist
compare with all text in website2.textlist
I realize that this solution could very quickly accumulate a lot of data, so it might only be possible to make it work with very small websites.
I haven't decided on the actual text comparison algorithm yet, but right now I'm more interested in getting the actual process algorithm working first.
I'm thinking it would be a good idea to extract all text as individual text pieces (from paragraphs, tables, headers and so on), as text can move around on pages.
I'm implementing this in C# (maybe ASP.NET).
I'm very interested in any input or advice you might have, so please shoot! :)
My approach to this problem would be to google for specific, fairly unique blocks of text whose copyright you are trying to protect.
Having said that, if you want to build your own solution, here are some comments:
Respect robots.txt. If they have marked the site as do-not-crawl, chances are they are not trying to profit from your content anyway.
You will need to refresh the site structure you have stored from time-to-time as websites change.
You will need to properly separate text from HTML tags and JavaScript.
You will essentially need to do a full text search in the entire text of the page (with tags/Script removed) for the text you wish to protect. There are good, published algorithms for this.
You're probably going to be more interested in fragment detection. for example, lots of pages will have the word "home" on them and you don't care. But it's fairly unlikely very many pages will have exactly the same words on the entire page. So you probably want to compare and report on pages that have exct matches of length 4,5,6,7,8, etc words and counts for each length. Assign a score and weight them and if you exceed your "magic number" report the suspected xeroxers.
For C#, you can use the webBrowser() to get a page and fairly easily get its text. Sorry, no code sample handy to copy/paste but MSDN usually has pretty good samples.
Related
I am currently developing a Word-Completion application in C# and after getting the UI up and running, keyboard hooks set, and other things of that nature, I came to the realization that I need a WordList. The only issue is, I cant seem to find one with the appropriate information. I also don't want to spend an entire week formatting and gathering a WordList by hand.
The information I want is something like "TheWord, The definition, verb/etc."
So, it hit me. Why not download a basic word list with nothing but words(Already did this; there are about 109,523 words), write a program that iterates through every word, connects to the internet, retrieves the data(definition etc) from some arbitrary site, and creates XML data from said information. It could be 100% automated, and I would only have to wait for maybe an hour depending on my internet connection speed.
This however, brought me to a few questions.
How should I connect to a site to look up these words? << This my actual question.
How would I read this information from the website?
Would I piss off my ISP or the website for that matter?
Is this a really bad idea? Lol.
How do you guys think I should go about this?
EDIT
Someone noticed that Dictionary.com uses the word as a suffix in the url. This will make it easy to iterate through the word file. I also see that the webpage is stored in XHTML(Or maybe just HTML). Here is the source for the Word "Cat". http://pastebin.com/hjZj6AC1
For what you marked as your actual question - you just need to download the data from the website and find what you need.
A great tool for this is CsQuery which allows you to use jquery selectors.
You could do something like this:
var dom = CQ.CreateFromUrl("http://www.jquery.com");
string definition = dom.Select(".definitionDiv").Text();
I am not sure how to start solving this problem so any suggestions will be of help.
My client has a number of static HTML pages running into hundreds of files. These under go updates every now and then and are overwritten on the website. We list these pages on the website via a simple left hand side explorer mimicking the folder structure in which these files are given to us.
We now want to give the ability to search these files and display matching results. Doing a brute search through such a large number of files is going to be very time consuming. Matching related words (for example plurals, misspellings etc) is also desirable. Showing results in the order of popularity would be a useful feature. I am not sure how to get started on this. Should we pre-process the html files after every update for instance? Any recommended indexing libraries available in .NET? What little programming has been done on the website has been done using C#.
Thanks
MS
Lucene.net may be of interest.
I´d first write a simple program to transfer all those files contents to a database. Then you could implement your search properly without having to read all files every time.
I am not sure if its within your budget, but Google can do it for you as user1161318 pointed out.
Try Google Site Search - http://www.google.co.uk/enterprise/search/products_gss.html
I need to highlight changes(diff) between 2 database text fields in a asp.net application.
I'm used to "beyond compare" text compare, so the Ideal solution will do something like it, but if it just highlight the differences, that would be OK.
http://www.scootersoftware.com/moreinfo.php?zz=screenshot&shot=TextCompare
The content is HTML, so if it could compare the HTML rendered text, it would be even better.
So, i need a link or a control (free if possible) to do that job.
thanks
I think what you're asking is pretty complex. I can't find any control out there that will do this, but i did find this: Difference Algorithm for C#
It doesn't give you exactly what you need, but it's a good place to start.
I've just come across this question and it looks like there are more recent options - eg. Html Diff algorithm for .NET
I am working on an application which requires matching of numbers from a scanned image file to database entry and update the database with the match result.
Say I have image- employee1.jpg. This image will have two two handwritten entries - Employee number and the amount to be paid to the employee. I have to read the employee number from the image and query the database for the that number, update the employee with the amount to be paid as got from the image. Both the employee number and amount to be paid are written inside two boxes at a specified place on the image.
Is there any way to automate this. Basically I want a solution in .net using c#. I know this can be done using artificial neural networks.
Any ideas would be much appreciated.
You can use Microsoft Office Document Imaging Library (MODI), which is contained in the Office 2003/2007.
Links:
OCR with Microsoft® Office - Code
Project - example of using MODI
Microsoft Office Document Imaging -
Wikipedia - contains a simple
example in VB.NET
Pattern recognition is a basic example when neural networks are studied. I don't know if is any library/framework to work with AI in C#. If you find one, first you have to do is to train the network (supervised learning) and for this you need to prepare a big sample set of images; more examples -> result more accurate. In the other hand you can use OpenCV (C/C++, Python and Java) that is a library specialized in computer vision and has a module to implement AI methods.
Have a nice day!
Oscar.
I think this is very hard to automate. The problem is just because you need some kind of very good OCR software. And even if you got this, what if it reads something wrong, cause of the frouzy handwriting of someone? If the ID is wrong the paid is booked to the wrong employee and if the amount is wrong he got the wrong salary!
Both are things you won't really happen. Just to show you how hard a good ocr to find is, just take a look out there on how a captcha works. The principle is nothing more than an image of an hard to read text.
So my opinion would be, that you can't really automate this process. At least you can write a program to assist a human by entering the values manually (also take a look at Amazon Mechanical Turk):
Show on the right the picture with the handwritten values, or if they are always on the same position or specially marked (with a box around them, etc.), try to find these places automatically and show them to the user.
On the left offer two textboxes, where the user can enter the values.
To get this to a fast and fluent process, you have to take great care about how the user can enter easily the values by just using the keyboard:
When showing a new picture, set the focus to the id textbox
If the user id is always a specific length, switch to the next box if all numbers are entered
(If you allow this, a backspace in the empty next box should focus back to the previous one)
Otherwise allow a change to the next textbox by hitting tab or return
Normally these textboxes are arranged above each other (not side by side), thus you should support switching between them using the up down arrow keys.
After finishing the entry in the last textbox automatically show the next image.
Also in this case of a fresh new entry (nothing already entered) allow a easy switch back to the old entry by using the backspace or left arrow key)
By using such a process a single person can enter many entries into your database and the costs are much cheaper than finding wrong entries in your database afterwards.
Just a last suggestion:
Cause this is a boring process for a human which can easily leads to errors, maybe let two people enter these values and only if both are entering the same than take this value as approved. This should lead to a correct rate somewhere above 99%. If you need absolutely 100% think about letting 4-5 people checking one entry and only if all of them enter the same values take it as approved. To get also a comparsion about how good your ocr software would be, just let it also run over your images and compare this results to the human entered values to get an idea, when you can really rely on your ocr only.
OCR engines are not trained to read handwritten text, so you might have trouble with MODI. You want to try to find an ICR engine. Even so, the best ones of these are only 80% accurate on good inputs. You might get better because you know that your text is always numbers.
This SO question/answer says that OCROpus has ICR
FOSS Intelligent Character Recognition (ICR)
There is LeadTools SDK for OCR/ICR. This is very handy in recognising the handwritten characters. I am doing a feasibilty study with this, and till now I think it will work out. leadTools have provided components which can be used in your application, it supports C, C++, C#, VB.Net etc.
You can visit the following link for this:
http://www.leadtools.com/downloads/default.htm?category=
In my web application I am developing a comment functionality, where user's can comment. But I am facing a problem which is I want to allow simple HTML tags in the comment box. HTML tags like <b>, <strong>, <i>, <em>, <u>, etc., that are normally allowed to enter in a commenting box. But then I also want when user presses enter then it will be automatically converted into breaks (<br /> tags) and get stored into database, so that when I'll display them in the web page then they'll look like as user entered.
Can you please tell me how to parse that user entered only allowed set of HTML tags and how to convert enters into <br /> tags and then store them in database.
Or if anyone have some better idea or suggestion to implement this kind of functionality. I am using ASP.NET 2.0 (C#)
I noticed that StackOverflow.com is doing the same thing on Profile Editing. When we edit our profile then below the "About Me" field "basic HTML allowed" line is written, I want to do almost the same functionality.
I don't have a C# specific answer for you, but you can go about it a few different ways. One is to let the user input whatever they want, then you run a filter over it to strip out the "bad" html. There are numerous open source filters that do this for PHP, Python, etc. In general, it's a pretty difficult problem, and it's best to let some well developed 3rd party code do this rather than write it yourself.
Another way to handle it is to allow the user to enter comments in some kind of simpler markup language like BBCode, Textile, or Markdown (stackoverflow is using Markdown), perhaps in conjunction with a nice Javascript editor. You then run the user's text through a processor for one of these markup languages to get the HTML. You can usually obtain implementations of these processors for whatever language you are using. These processors usually strip out the "bad" HTML.
Its rather "simple" to do that in php and python due to the large number of functions.I am still learning c# .lol. but havent yet come across the function.The chances are that it exists and all you need to do is search for it.I mean a function that can take the user input,search for the allowed tags (which are in an array of course) and replace the <> with something else like [] then use a function to escape the other html tags.In php we use htmlentities().
Something like
<code>
$txt=$_POST['comment'];
$txt=strreplace("<b>*</b>","[b]*[/b],"$txt");
$securetxt=htmlentities($txt);
$finaltxt=strreplace("[b]*[/b]","<b>*</b>","$securetxt");
//Now save to Db
I'm not sure, but I think you have to escape html characters when inserting in database and when retrieving echo them unescaped, so the browser can see it just like html.
I don´t know asp.net, but in php there´s an easy function, strip_tags, that let you add exceptions (in your case, b, em, etc.). If there´s nothing like that in C# you can write a regular expression that strips out all tags except the allowed ones but chances are that such an expression already exists so it should be easy to find.
replacing \n (or something similar) with br shouldn´t be a problem either with a simple search and replace.
This is a dangerous road to go down. You might think you can do some awesome regexes, or find someone who can help you with it, but sanitizing SOME markup and leaving other is just crazy talk.
I highly recommend you look into BBCode or another token system. Even something untokenized such as what SO uses, is probably a much better solution.