Consider:
<div>Anirudha Web blog</div>
What is the regular expression to get http://anirudhagupta.blogspot.com/
from the following?
<div>Anirudha Web blog</div>
If you suggest something in C# that's good. I also like jQuery to do this.
If you want to use jQuery you can do the following.
$('a').attr('href')
Quick and dirty:
href="(.*?)"
Ok, let's go with another regex for parsing URLs. This comes from RFC 2396 - URI Generic Syntax: Parsing a URI Reference with a Regular Expression
^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\?([^#]*))?(#(.*))?
Of course, you can have relative URL address into your HTML code, you'll need to address them in another way; I can recommend you to use C# Uri Constructor (Uri, String).
The simplest way to do this is using the following regular expression.
/href="([^"]+)"/
This will get all characters from the first quote until it finds a character that is a quote. This is, in most languages, the fastest way to get a quoted string, that can't itself contain quotes. Quotes should be encoded when used in attributes.
UPDATE: A complete Perl program for parsing URLs would look like this:
use 5.010;
while (<>) {
push #matches, m/href="([^"]+)"/gi;
push #matches, m/href='([^']+)'/gi;
push #matches, m/href=([^"'][^>\s]*)[>\s]+/gi;
say for #matches;
}
It reads from stdin and prints all URLs. It takes care of the three possible quotes. Use it with curl to find all the URLs in a webpage:
curl url | perl urls.pl
The right way to do this is to load the HTML into the C# XML parser and then use XPath to query the URLs. This way you don't have to worry about parsing at all.
You don't need a complicated regular expression or HTML parser, since you only want to extract links. Here's a generic way to do it.
data="""
<html>
abcd ef ....
blah blah <div>Anirudha Web blog</div>
blah ...
<div><a href="http://mike.blogspot.com/">Mike's Web blog
</a></div>
end...
</html>
"""
for item in data.split("</a>"):
if "<a href" in item:
start_of_href = item.index("<a href") # get where <a href=" is
print item[start_of_href+len('<a href="'):] # print substring from <a href onwards.
The above is Python code, but the idea behind you can adapt in your C# language. Split your HTML string using "</a>" as delimiter. Go through each split field, check for "href", then get the substr after "href". That will be your links.
Related
I am trying to essentially write an outbound URL matcher so I can replace a stream of html containing URL's to point to my CDN. I cant use the IIS URL Rewrite module as I am using compression. I currently have a regex that matches on a sub folder for a specific file type i.e.
Regex ASSET_PATH = new Regex(#"(?i)assets/([A-Za-z0-9\-_/.]+)\.(jpg|jpeg|bmp|tiff|png|gif|js|css|mov|mp4|ogg|avi|mp3)", RegexOptions.Compiled | RegexOptions.CultureInvariant | RegexOptions.IgnoreCase );
This works great and allows me to manipulate anything in the string from that point onwards ( i.e. from "assets/" onwards to the right ). What I need to achieve is to manipulate the string to the left of the "assets/" sub-folder, without necessarily knowing the format? Here are some examples :
<img src="./assets/123/pig.jpg" />
<img src="http://mysite.blah/assets/123/pig.jpg" />
<img src="http://www.mysite.blah/assets/123/pig.jpg" />
<img src='assets/123/pig.jpg' />
in css / inline styles :
background-image : URL('assets/123/pig.jpg')
background-image : URL(http://www.mysite.blah/assets/123/pig.jpg)
anyway, I think you get the picture. I essentially want to be able to look to the "left" of the word "assets" until I can find the logical start point of the url and then manipulate it from there to point to my CDN.
I'm not sure this is possible in regex, so any suggestions using a combination of regex / c# /HTML Agility Pack are welcome
Is this what you're after?
(?<BeforeAssets>.*?(?:\/|^))assets\/(?<AfterAssets>[A-Za-z0-9\-_\/.]+)\.(?<FileExtension>jpg|jpeg|bmp|tiff|png|gif|js|css|mov|mp4|ogg|avi|mp3)
You can try this out here: http://regexstorm.net/tester
Or here: https://regex101.com/r/b8XxcF/1
NB: In the above regex I escaped the forward slash characters. .Net doesn't require this, but doesn't complain; and doing so makes this compatible with other Regex engines; which means it can be tested on Regex101.
When testing with those tools you'll need to specify the MultiLine or SingleLine options to get the example where assets/ has nothing preceding it, since otherwise the ^ character won't match the start of that line. This option may not be required in your code; i.e. if you're only matching one string at a time, rather than a whole block of text.
Update
Apologies for misreading; you're parsing the full HTML page; not just the URIs returned from that page. To do this you could use something like:
["'\(](?<BeforeAssets>[^"'\(\)]*?)assets\/(?<AfterAssets>[A-Za-z0-9\-_\/.]+)\.(?<FileExtension>jpg|jpeg|bmp|tiff|png|gif|js|css|mov|mp4|ogg|avi|mp3)
(thankfully characters ", ', and ( are illegal in the URL, so should be OK to detect the start of a variable: https://www.rfc-editor.org/rfc/rfc3986#section-2.2.)
This isn't fool-proof; it's better to use an HTML parsing tool, then pull out the URIs from that; but if you are doing everything with regex, hopefully this will help.
I have a source to a web page and I need to extract the body. So anything between </head><body> and </body></html>.
I've tried the following with no success:
var match = Regex.Match(output, #"(?<=\</head\>\<body\>)(.*?)(?=\</body\>\</html\>)");
It finds a string but cuts it off long before </body></html>. I escaped characters based on the RegEx cheat sheet.
What am i missing?
I'd recommend using the HtmlAgilityPack instead - parsing HTML with regular expressions is very, very fragile.
The latest version even supports Linq so you can get your content like this:
HtmlWeb web = new HtmlWeb();
HtmlDocument doc = web.Load("http://stackoverflow.com");
string html = doc.DocumentNode.Descendants("body").Single().InnerHtml;
Regex is not meant for such html handling, as many here would say. Without having your sample web page / html, I can only say that try removing the non-greedy ? quantifier in (.*?) and try. After all, a html page will have only one head and body.
Though regexes are definitely not the best tool for this task, there are a few suggestions and points I would like to make:
un-escape the angle brackets - with the # before your string, they are going through to the regex and they do not need to be escaped for a .NET regex
with your regex, you need to make sure that the head/body tag combinations do not have any white-space between them.
with your regex, the body tag cannot have any attributes.
I would suggest something more like:
(?<=</head>\s*<body(\s[^>]*)?>)(.*?)(?=</body>\s*</html>)
this seems to work for me on the source of this page!
As the others have said, the correct way to handle this is with an HTML-specific tool. I just want to point out some problems with that cheat-sheet.
First, it's wrong about angle brackets: you do not need to escape them. In fact, it's wrong twice: it also says \< and \> match word boundaries, which is both incorrect for .NET, and incompatible with the advice about escaping angle brackets.
That cheat-sheet is just a random collection of regex syntax elements; most of them will work in most flavors, but many are guaranteed not to work in your particular flavor, whatever it happens to be. I recommend you disregard it and rely instead on .NET-specific documents or Regular-Expressions.info. The books Mastering Regular Expressions and Regular Expressions Cookbook are both excellent, too.
As for your regex, I don't see how it could behave the way you say it does. If it were going to fail, I would expect it to fail completely. Does your HTML document contain a CDATA section or SGML comment with </body></html> inside it? Or is it really two or more HTML documents run together?
In my C# program I wrote a Google Search Function, which works by fetching the source from each page and getting the URLs via regex.
My actual Regex is:
(?:(?:(?:http)://)(?:w{3}\\.)?(?:[a-zA-Z0-9/;\\?&=:\\-_\\$\\+!\\*'\\(\\|\\\\~\\[\\]#%\\.])+)
This works good at the moment, but I get for example URLs like http://www.example.com/forums/arcade.php?efdf=332
I just want to get in this case the URL without the ?efdf=332 at the end.
So how should I change the regex?
http://(?:www\.)?[a-zA-Z0-9/;&=:_$+!*'()|~\[\]#%.\\-]+
does the same as your regex (I've removed a lot of unnecessary cruft) but stops matching a link before a ?.
In C#:
Regex regexObj = new Regex(#"http://(?:www\.)?[a-zA-Z0-9/;&=:_$+!*'()|~\[\]#%.\\-]+")
That said, I'm not sure this is such a good way of matching URLs (what about https, ftp, mailto etc.?)
You can use the Uri class to access various parts of the URL and either remove the query string from the end, or concatenate the parts you want.
I have an ASP.NET/C# application, part of which converts WWW links to mailto links in an HTML email.
For example, if I have a link such as:
www.site.com
It gets rewritten as:
mailto:my#address.com?Subject=www.site.com
This works extremely well, until I run into URLs with ampersands, which then causes the subject to be truncated.
For example the link:
www.site.com?val1=a&val2=b
Shows up as:
mailto:my#address.com?Subject=www.site.com?val1=a&val2=b
Which is exactly what I want, but then when clicked, it creates a message with:
subject=www.site.com?val1=a
Which has dropped the &val2, which makes sense as & is the delimiter in a mailto command.
So, I have tried various other was to work around this with no success.
I have tried implicitly quoting the subject='' part and that did nothing.
I (in C#) replace '&' with & which Live Mail and Thunderbird just turn back into:
www.site.com?val1=a&val2=b
I replaced '&' with '%26' which resulted in:
mailto:my#address.com?Subject=www.site.com?val1=a%26amp;val2=b
In the mail with the subject:
www.site.com?val1=a&val2=b
EDIT:
In response to how URL is being built, this is much trimmed down but is the gist of it. In place of the att.Value.Replace I have tried System.Web.HtmlUtility.URLEncode calls which also results in a failure
HtmlAgilityPack.HtmlNodeCollection nodes =doc.DocumentNode.SelectNodes("//a[#href]");
foreach (HtmlAgilityPack.HtmlNode link in nodes)
{
HtmlAgilityPack.HtmlAttribute att = link.Attributes["href"];
att.Value = att.Value.Replace("&", "%26");
}
Try mailto:my#address.com?Subject=www.site.com?val1=a%26val2=b
& is an HTML escape code, whereas %26 is a URL escape code. Since it's a URL, that's all you need.
EDIT: I figured that's how you were building your URL. Don't build URLs that way! You need to get the %26 in there before you let anything else parse or escape it. If you really must do it this way (which you really should try to avoid), then you should search for "&" instead of just "&" because the string has already been HTML escaped at this point.
So, ideally, you build your URL properly before it's HTML escaped. If you can't do it properly, at least search for the right string instead of the wrong one. "&" is the wrong one.
You cant put any character as subject. You could try using System.Web.HttpUtility.URLEncode function on the subject´s value...
Using the URL escape code %26 is the right way.
Sadly this is still not working on the Android OS because of bug 8023
What I ended up doing for my case was eliminating the &.
www.site.com/mytest.php?val1=a=b=c. Where the 2nd and 3rd = would be equivalent to www.site.com?val1=a&val2=b&val3=c
In mytest.php I explode on ? and then explode again on =.
A total hack I know but it does work for me.
How can I write a regular expression to replace links with no link text like this:
with
http://www.somesite.com
?
This is what I was trying to do to capture the matches, and it isn't catching any. What am I doing wrong?
string pattern = "<a\\s+href\\s*=\\s*\"(?<href>.*)\">\\s*</a>";
I wouldn't use a regex - I'd use the Html Agility Pack, and a query like:
foreach(HtmlNode link in doc.DocumentElement.SelectNodes("//a[.='']")) {
link.InnerText = link.GetAttribute("href");
}
I could be wrong, but I think you simply need to change the quantifier within the href group to be lazy rather than greedy.
string pattern = #"<a\s+href\s*=\s*""(?<href>.*?)"">\s*</a>";
(I've also changed the type of the string literal to use #, for better readability.)
The rest of the regex appears fine to me. That you're not capturing any matches at all makes me think otherwise, but there could be a problem in the rest of the code (or even the input data - have you verified that?).
I would suggest
string pattern = "(<a\\b[^>]*href=\"([^\"]+)\"[^>]*>)[\\s\\r\\n]*(</a>)";
This way also links with their href attribute somewhere else would be captured.
Replace with
"$1$2$3"
The usual word of warning: HTML and regex are essentially incompatible. Use with caution, this might blow up.
Marc Gravell has the right answer, regexes are fundamentally bad at parsing HTML (see Can you provide some examples of why it is hard to parse XML and HTML with a regex? for why). See Can you provide an example of parsing HTML with your favorite parser? for examples using a variety of parsers.