Why Is The Imgur API slow compared to the website - c#

public sealed class ImgurUpload
{
public event EventHandler<UploadCompleteEventArgs> UploadComplete;
public void PostToImgur(string location, string key, string name = "", string caption = "")
{
try
{
using (var webClient = new WebClient())
{
NameValueCollection values = new NameValueCollection
{
{ "image", ConvertToBase64(location) },
{ "key", key },
{ "name", name },
{ "caption", caption}
};
webClient.UploadValuesAsync(new Uri("http://api.imgur.com/2/upload.xml"), "POST", values);
webClient.UploadValuesCompleted += ((sender, eventArgs) =>
{
byte[] response = eventArgs.Result;
XDocument result = XDocument.Load(XmlReader.Create(new MemoryStream(response)));
if (UploadComplete != null)
UploadComplete(this, new UploadCompleteEventArgs(result));
});
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private string ConvertToBase64(string imageLocation)
{
byte[] imageData = null;
using (FileStream fileStream = File.OpenRead(imageLocation))
{
imageData = new byte[fileStream.Length];
fileStream.Read(imageData, 0, imageData.Length);
}
return Convert.ToBase64String(imageData);
}
}
public class UploadCompleteEventArgs : EventArgs
{
public string Original { get; private set; }
public string ImgurPage { get; private set; }
public string DeletePage { get; private set; }
public UploadCompleteEventArgs(XDocument xmlDoc)
{
var objLinks = from links in xmlDoc.Descendants("links")
select new
{
original = links.Element("original").Value,
imgurPage = links.Element("imgur_page").Value,
deletePage = links.Element("delete_page").Value
};
Original = objLinks.FirstOrDefault().original;
ImgurPage = objLinks.FirstOrDefault().imgurPage;
DeletePage = objLinks.FirstOrDefault().deletePage;
}
}
Above is a class I wrote to upload an image to imgur using the Anonymous API. I have used the API in the past and have always found it to be considerably slower than the website uploader and slower than other .NET applications that use web requests to effectively send data to the website directly rather than using the API.
I posted the full class above as it may be something I'm doing (or not doing) that's causing the issue. I'd really appreciate it if anybody can identify the issue for me.
I did some fair testing earlier today and one result for example, is as followed:
800kb image via the imgur website = 35 seconds
800kb image via using my class = 1minute 20 seconds

The one you are uploading is ~35% bigger because you're uploading it as a STRING.
Upload via bytes and it should be just as fast.

Related

How to upload any file on Slack via Slack-App in c#

I need help with uploading files to Slack.
I have a Slack-App that is working with my code(below) so far. But all I can do is post messages. I can not attach images to the messages - because I do not understand how to use the so called "methods" and the syntax Slack is "showing" on their API-page.
This creates my "content" and below its just a Stream for reading a file I could upload:
public class PostMessage
{
public FormUrlEncodedContent Content(string message, string file)
{
var values = new Dictionary<string, string>
{
{"token", "xoxp-myToken"},
{ "username", "X"},
{ "channel", "myChannel"},
{ "as_user", "false"},
{"text", message},
{ "content", file},
{ "attachments","[{ \"fallback\":\"dummy\", \"text\":\"this is a waste of time\"}]"}
};
var content = new FormUrlEncodedContent(values);
return content;
}
}
public class PostFile
{
String path = #"C:\Users\f.held\Desktop\Held-Docs\dagged.jpg";
public string ReadImageFile()
{
FileInfo fileInfo = new FileInfo(path);
long imageFileLength = fileInfo.Length;
FileStream fs = new FileStream(path, FileMode.Open, FileAccess.Read);
BinaryReader br = new BinaryReader(fs);
byte[] imageData = br.ReadBytes((int)imageFileLength);
var str = Encoding.Default.GetString(imageData);
return str;
}
}
}
The client that communicates:
public class SlackClient
{
private readonly Uri _webhookUrl;
private readonly HttpClient _httpClient = new HttpClient {};
public SlackClient(Uri webhookUrl)
{
_webhookUrl = webhookUrl;
}
public async Task<HttpResponseMessage> SendMessageAsync(FormUrlEncodedContent content)
{
var response = await _httpClient.PostAsync(_webhookUrl, content);
return response;
}
}
}
The Main:
public static void Main(string[] args)
{
Task.WaitAll(IntegrateWithSlackAsync());
}
private static async Task IntegrateWithSlackAsync()
{
var webhookUrl = new Uri("https://slack.com/api/files.upload");
var slackClient = new SlackClient(webhookUrl);
PostMessage PM = new PostMessage();
PostFile PF = new PostFile();
while (true)
{
Console.Write("Type a message: ");
var message = Console.ReadLine();
var testFile = PF.ReadImageFile();
FormUrlEncodedContent payload = PM.Content(message, testFile);
var response = await slackClient.SendMessageAsync(payload);
var isValid = response.IsSuccessStatusCode ? "valid" : "invalid";
Console.WriteLine($"Received {isValid} response.");
Console.WriteLine(response);
response.Dispose();
}
}
}
}
If somebody has an example on what a upload has to look like. Or even better,
if somebody could really explain the syntax these Slack-Messages have to have.
That would be great! I still do not know where and HOW I should put the so called
"Accepted content types: multipart/form-data, application/x-www-form-urlencoded" to my upload. I just can not find examples on this...
Edit:
What confuses me needlesly is that Slack states they have an extra method called file.upload - but we shouldn't use it anymore, we should use just postMessage.
But how would I "pack" a file in a message? My syntax always seems to be off. Especially when it comes to "content"...
I just can not figure out what the c#-code has to look like. Where do I declare the aforementioned "content type"?
Another problem is, it always sends my messages through - means I get a 200-response from the server. But it never shows the file (which probably means the syntax is off) Or I get the 200-response but the message never shows in Slack.
Images in message
If you want to include an image in your message (along with some text) you can do so by adding images as message attachment to a normal message send with chat.postMessage.
For that you need a public URL of your image and that link with the image_url property to an attachment. That attachment can also contain text, and you can add multiple attachments to your message.
This is how it looks like:
And here is how this message looks in JSON:
{
"channel": "test",
"text": "This is a message example with images in the attachment",
"attachments": [
{
"fallback": "game over",
"text": "This is some text in the attachement",
"image_url": "https://i.imgur.com/jO9N3eJ.jpg"
}
]
}
Uploading images
The image URL needs to be publicly accessible on the Internet. So you need to host your image file on a public webserver or upload it to a image cloud service (e.g. imgur.com).
You can also use Slack as cloud service for your images. Here is how that works:
Upload to Slack: Upload your image to your Slack workspace with files.upload
Get public URL: Get a public URL for your image file with files.sharedPublicURL. Normally all files on Slack are private, but you can only use public URLs for message attachments.
Send message: Include your image as attachment in a message: Use the permalink_public property of your image file as value for image_url
Example code
Here is a full working example in C# for first uploading an image to Slack and then using it in a message.
Note: This example requires Newtonsoft.Json.
using System;
using System.Net;
using System.Collections.Specialized;
using System.Text;
using Newtonsoft.Json;
public class SlackExample
{
// classes for converting JSON respones from API method into objects
// note that only those properties are defind that are needed for this example
// reponse from file methods
class SlackFileResponse
{
public bool ok { get; set; }
public String error { get; set; }
public SlackFile file { get; set; }
}
// a slack file
class SlackFile
{
public String id { get; set; }
public String name { get; set; }
public String permalink_public { get; set; }
}
// reponse from message methods
class SlackMessageResponse
{
public bool ok { get; set; }
public String error { get; set; }
public String channel { get; set; }
public String ts { get; set; }
}
// a slack message attachment
class SlackAttachment
{
public String fallback { get; set; }
public String text { get; set; }
public String image_url { get; set; }
}
// main method with logic
public static void Main()
{
String token = "xoxp-YOUR-TOKEN";
/////////////////////
// Step 1: Upload file to Slack
var parameters = new NameValueCollection();
// put your token here
parameters["token"] = token;
var client1 = new WebClient();
client1.QueryString = parameters;
byte[] responseBytes1 = client1.UploadFile(
"https://slack.com/api/files.upload",
"C:\\Temp\\Stratios_down.jpg"
);
String responseString1 = Encoding.UTF8.GetString(responseBytes1);
SlackFileResponse fileResponse1 =
JsonConvert.DeserializeObject<SlackFileResponse>(responseString1);
String fileId = fileResponse1.file.id;
/////////////////////
// Step 2: Make file public and get the URL
var parameters2 = new NameValueCollection();
parameters2["token"] = token;
parameters2["file"] = fileId;
var client2 = new WebClient();
byte[] responseBytes2 = client2.UploadValues("https://slack.com/api/files.sharedPublicURL", "POST", parameters2);
String responseString2 = Encoding.UTF8.GetString(responseBytes2);
SlackFileResponse fileResponse2 =
JsonConvert.DeserializeObject<SlackFileResponse>(responseString2);
String imageUrl = fileResponse2.file.permalink_public;
/////////////////////
// Step 3: Send message including freshly uploaded image as attachment
var parameters3 = new NameValueCollection();
parameters3["token"] = token;
parameters3["channel"] = "test_new";
parameters3["text"] = "test message 2";
// create attachment
SlackAttachment attachment = new SlackAttachment();
attachment.fallback = "this did not work";
attachment.text = "this is anattachment";
attachment.image_url = imageUrl;
SlackAttachment[] attachments = { attachment };
parameters3["attachments"] = JsonConvert.SerializeObject(attachments);
var client3 = new WebClient();
byte[] responseBytes3 = client3.UploadValues("https://slack.com/api/chat.postMessage", "POST", parameters3);
String responseString3 = Encoding.UTF8.GetString(responseBytes3);
SlackMessageResponse messageResponse =
JsonConvert.DeserializeObject<SlackMessageResponse>(responseString3);
}
}
Here is a shorter working example showing how to just upload any file to Slack with C# only. The example will also automatically share the file the given channel.
I have included the logic to convert the API response from JSON, which will always be needed to determine if the API call was successful.
Note: This example requires Newtonsoft.Json
using System;
using System.Net;
using System.Collections.Specialized;
using System.Text;
using Newtonsoft.Json;
public class SlackExample
{
// classes for converting JSON respones from API method into objects
// note that only those properties are defind that are needed for this example
// reponse from file methods
class SlackFileResponse
{
public bool ok { get; set; }
public String error { get; set; }
public SlackFile file { get; set; }
}
// a slack file
class SlackFile
{
public String id { get; set; }
public String name { get; set; }
}
// main method with logic
public static void Main()
{
var parameters = new NameValueCollection();
// put your token here
parameters["token"] = "xoxp-YOUR-TOKEN";
parameters["channels"] = "test";
var client = new WebClient();
client.QueryString = parameters;
byte[] responseBytes = client.UploadFile(
"https://slack.com/api/files.upload",
"D:\\temp\\Stratios_down.jpg"
);
String responseString = Encoding.UTF8.GetString(responseBytes);
SlackFileResponse fileResponse =
JsonConvert.DeserializeObject<SlackFileResponse>(responseString);
}
}
About content types: Those are part of the header of a HTTP request and can be set manually in the WebClient object (see also this answer). However, for our case you can ignore it, because the default content types that WebClient is using for the POST request will work just fine.
Also see this answer on how to upload files with the WebClient class.
Here is another complete example for uploading a file to Slack, this time using the async approach with HttpClient.
Note: This example requires Newtonsoft.Json.
using Newtonsoft.Json;
using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
namespace SlackExample
{
class UploadFileExample
{
private static readonly HttpClient client = new HttpClient();
// classes for converting JSON respones from API method into objects
// note that only those properties are defind that are needed for this example
// reponse from file methods
class SlackFileResponse
{
public bool ok { get; set; }
public String error { get; set; }
public SlackFile file { get; set; }
}
// a slack file
class SlackFile
{
public String id { get; set; }
public String name { get; set; }
}
// sends a slack message asynchronous
// throws exception if message can not be sent
public static async Task UploadFileAsync(string token, string path, string channels)
{
// we need to send a request with multipart/form-data
var multiForm = new MultipartFormDataContent();
// add API method parameters
multiForm.Add(new StringContent(token), "token");
multiForm.Add(new StringContent(channels), "channels");
// add file and directly upload it
FileStream fs = File.OpenRead(path);
multiForm.Add(new StreamContent(fs), "file", Path.GetFileName(path));
// send request to API
var url = "https://slack.com/api/files.upload";
var response = await client.PostAsync(url, multiForm);
// fetch response from API
var responseJson = await response.Content.ReadAsStringAsync();
// convert JSON response to object
SlackFileResponse fileResponse =
JsonConvert.DeserializeObject<SlackFileResponse>(responseJson);
// throw exception if sending failed
if (fileResponse.ok == false)
{
throw new Exception(
"failed to upload message: " + fileResponse.error
);
}
else
{
Console.WriteLine(
"Uploaded new file with id: " + fileResponse.file.id
);
}
}
static void Main(string[] args)
{
// upload this file and wait for completion
UploadFileAsync(
"xoxp-YOUR-TOKEN",
"C:\\temp\\Stratios_down.jpg",
"test"
).Wait();
Console.ReadKey();
}
}
}

How to Loop calls to Pagination URL in C# HttpClient to download all Pages from JSON results

My 1st question, so please be kind... :)
I'm using the C# HttpClient to invoke Jobs API Endpoint.
Here's the endpoint: Jobs API Endpoint (doesn't require key, you can click it)
This gives me JSON like so.
{
"count": 1117,
"firstDocument": 1,
"lastDocument": 50,
"nextUrl": "\/api\/rest\/jobsearch\/v1\/simple.json?areacode=&country=&state=&skill=ruby&city=&text=&ip=&diceid=&page=2",
"resultItemList": [
{
"detailUrl": "http:\/\/www.dice.com\/job\/result\/90887031\/918715?src=19",
"jobTitle": "Sr Security Engineer",
"company": "Accelon Inc",
"location": "San Francisco, CA",
"date": "2017-03-30"
},
{
"detailUrl": "http:\/\/www.dice.com\/job\/result\/cybercod\/BB7-13647094?src=19",
"jobTitle": "Platform Engineer - Ruby on Rails, AWS",
"company": "CyberCoders",
"location": "New York, NY",
"date": "2017-04-16"
}
]
}
I've pasted a complete JSON snippet so you can use it in your answer. The full results are really long for here.
Here's are the C# classes.
using Newtonsoft.Json;
using System.Collections.Generic;
namespace MyNameSpace
{
public class DiceApiJobWrapper
{
public int count { get; set; }
public int firstDocument { get; set; }
public int lastDocument { get; set; }
public string nextUrl { get; set; }
[JsonProperty("resultItemList")]
public List<DiceApiJob> DiceApiJobs { get; set; }
}
public class DiceApiJob
{
public string detailUrl { get; set; }
public string jobTitle { get; set; }
public string company { get; set; }
public string location { get; set; }
public string date { get; set; }
}
}
When I invoke the URL using HttpClient and deserialize using JSON.NET, I do get the data back properly.
Here's the code I am calling from my Console App's Main method (hence the static list, I think this could be better refactored??)
private static List<DiceApiJob> GetDiceJobs()
{
HttpClient httpClient = new HttpClient();
var jobs = new List<DiceApiJob>();
var task = httpClient.GetAsync("http://service.dice.com/api/rest/jobsearch/v1/simple.json?skill=ruby")
.ContinueWith((taskwithresponse) =>
{
var response = taskwithresponse.Result;
var jsonString = response.Content.ReadAsStringAsync();
jsonString.Wait();
var result = JsonConvert.DeserializeObject<DiceApiJobWrapper>(jsonString.Result);
if (result != null)
{
if (result.DiceApiJobs.Any())
jobs = result.DiceApiJobs.ToList();
if (result.nextUrl != null)
{
//
// do this GetDiceJobs again in a loop? How?? Any other efficient elegant way??
}
}
});
task.Wait();
return jobs;
}
But now, how do I check if there are more jobs using the nextUrl field? I know I can check to see if it's not null, and if if not, that means there are more jobs to pull down.
Results from my debugging and stepping through
How do I do this recursively, and without hanging and with some delays so I don't cross the API limits? I think I have to use TPL ( Task Parallel Library) but am quite baffled.
Thank you!
~Sean
If you are concerned about response time of your app and would like to return some results before you actually get all pages/data from the API, you could run your process in a loop and also give it a callback method to execute as it gets each page of data from the API.
Here is a sample:
public class Program
{
public static void Main(string[] args)
{
var jobs = GetDiceJobsAsync(Program.ResultCallBack).Result;
Console.WriteLine($"\nAll {jobs.Count} jobs displayed");
Console.ReadLine();
}
private static async Task<List<DiceApiJob>> GetDiceJobsAsync(Action<DiceApiJobWrapper> callBack = null)
{
var jobs = new List<DiceApiJob>();
HttpClient httpClient = new HttpClient();
httpClient.BaseAddress = new Uri("http://service.dice.com");
var nextUrl = "/api/rest/jobsearch/v1/simple.json?skill=ruby";
do
{
await httpClient.GetAsync(nextUrl)
.ContinueWith(async (jobSearchTask) =>
{
var response = await jobSearchTask;
if (response.IsSuccessStatusCode)
{
string jsonString = await response.Content.ReadAsStringAsync();
var result = JsonConvert.DeserializeObject<DiceApiJobWrapper>(jsonString);
if (result != null)
{
// Build the full list to return later after the loop.
if (result.DiceApiJobs.Any())
jobs.AddRange(result.DiceApiJobs.ToList());
// Run the callback method, passing the current page of data from the API.
if (callBack != null)
callBack(result);
// Get the URL for the next page
nextUrl = (result.nextUrl != null) ? result.nextUrl : string.Empty;
}
}
else
{
// End loop if we get an error response.
nextUrl = string.Empty;
}
});
} while (!string.IsNullOrEmpty(nextUrl));
return jobs;
}
private static void ResultCallBack(DiceApiJobWrapper jobSearchResult)
{
if (jobSearchResult != null && jobSearchResult.count > 0)
{
Console.WriteLine($"\nDisplaying jobs {jobSearchResult.firstDocument} to {jobSearchResult.lastDocument}");
foreach (var job in jobSearchResult.DiceApiJobs)
{
Console.WriteLine(job.jobTitle);
Console.WriteLine(job.company);
}
}
}
}
Note that the above sample allows the callback method to access each page of data as it is received by the GetDiceJobsAsync method. In this case, the console, displays each page as it becomes available. If you do not want the callback option, you can simply pass nothing to GetDiceJobsAsync.
But the GetDiceJobsAsync also returns all the jobs when it completes. So you can choose to act on the whole list at the end of GetDiceJobsAsync.
As for reaching API limits, you can insert a small delay within the loop, right before you repeat the loop. But when I tried it, I did not encounter the API limiting my requests so I did not include it in the sample.

CefSharp custom SchemeHandler

Iam using CefSharp's SchemeHandler in order to grab resources from my C# project like .css, .js or .png files using a custom url for example custom://cefsharp/assets/css/style.css
I've 2 custom classes in order to archive this.
First class, MyCustomSchemeHandlerFactory will be the one that handles the custom Scheme and it looks like this, where "custom" will be the custom scheme:
internal class MyCustomSchemeHandlerFactory : ISchemeHandlerFactory
{
public const string SchemeName = "custom";
public IResourceHandler Create(IBrowser browser, IFrame frame, string schemeName, IRequest request)
{
return new MyCustomSchemeHandler();
}
}
The next class I've implemented is MyCustomSchemeHandler which will receive the call and output a response and it looks like this:
internal class MyCustomSchemeHandler : IResourceHandler
{
private static readonly IDictionary<string, string> ResourceDictionary;
private string mimeType;
private MemoryStream stream;
static MyCustomSchemeHandler()
{
ResourceDictionary = new Dictionary<string, string>
{
{ "/home.html", Properties.Resources.index},
{ "/assets/css/style.css", Properties.Resources.style}
};
}
public Stream Stream { get; set; }
public int StatusCode { get; set; }
public string StatusText { get; set; }
public string MimeType { get; set; }
public NameValueCollection Headers { get; private set; }
public Stream GetResponse(IResponse response, out long responseLength, out string redirectUrl)
{
redirectUrl = null;
responseLength = -1;
response.MimeType = MimeType;
response.StatusCode = StatusCode;
response.StatusText = StatusText;
response.ResponseHeaders = Headers;
var memoryStream = Stream as MemoryStream;
if (memoryStream != null)
{
responseLength = memoryStream.Length;
}
return Stream;
}
public bool ProcessRequestAsync(IRequest request, ICallback callback)
{
// The 'host' portion is entirely ignored by this scheme handler.
var uri = new Uri(request.Url);
var fileName = uri.AbsolutePath;
string resource;
if (ResourceDictionary.TryGetValue(fileName, out resource) && !string.IsNullOrEmpty(resource))
{
var resourceHandler = ResourceHandler.FromString(resource);
stream = (MemoryStream)resourceHandler.Stream;
var fileExtension = Path.GetExtension(fileName);
mimeType = ResourceHandler.GetMimeType(fileExtension);
callback.Continue();
return true;
}
else
{
callback.Dispose();
}
return false;
}
void GetResponseHeaders(IResponse response, out long responseLength, out string redirectUrl)
{
responseLength = stream == null ? 0 : stream.Length;
redirectUrl = null;
response.StatusCode = (int)HttpStatusCode.OK;
response.StatusText = "OK";
response.MimeType = mimeType;
}
bool ReadResponse(Stream dataOut, out int bytesRead, ICallback callback)
{
//Dispose the callback as it's an unmanaged resource, we don't need it in this case
callback.Dispose();
if (stream == null)
{
bytesRead = 0;
return false;
}
//Data out represents an underlying buffer (typically 32kb in size).
var buffer = new byte[dataOut.Length];
bytesRead = stream.Read(buffer, 0, buffer.Length);
dataOut.Write(buffer, 0, buffer.Length);
return bytesRead > 0;
}
bool CanGetCookie(Cookie cookie)
{
return true;
}
bool CanSetCookie(Cookie cookie)
{
return true;
}
void Cancel()
{
}
}
Inside this class I've defined a custom resource dictionary which will dictate what file from the resources will be used, so as I stated in the first example, custom://cefsharp/assets/css/style.css should load the resource Properties.Resources.style, the problem is that nothing gets loaded once I enter to the specific url, I've tried to output the mimeType and It works but somehow the file itself won't output correctly. Is there something wrong with my implementation?
Additionaly I've tried to output the raw file in the form of:
if (ResourceDictionary.TryGetValue(fileName, out resource) && !string.IsNullOrEmpty(resource))
{
MessageBox.Show(resource);
}
And it outputs the correct file without any problems.
To load the custom Scheme I use the following code before initializing CefSharp:
var settings = new CefSettings();
settings.RegisterScheme(new CefCustomScheme
{
SchemeName = MyCustomSchemeHandlerFactory.SchemeName,
SchemeHandlerFactory = new MyCustomSchemeHandlerFactory()
});
The above classes were based on the following links:
MyCustomSchemeHandlerFactory: FlashResourceHandlerFactory.cs
MyCustomSchemeHandler: CefSharpSchemeHandler.cs and ResourceHandler.cs
Since Cefsharp changed a bit in last few months here is an updated and easier way of handling 'file' protocol. I wrote blog post on this matter.
What you want to add is your scheme handler and its factory:
using System;
using System.IO;
using CefSharp;
namespace MyProject.CustomProtocol
{
public class CustomProtocolSchemeHandler : ResourceHandler
{
// Specifies where you bundled app resides.
// Basically path to your index.html
private string frontendFolderPath;
public CustomProtocolSchemeHandler()
{
frontendFolderPath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "./bundle/");
}
// Process request and craft response.
public override bool ProcessRequestAsync(IRequest request, ICallback callback)
{
var uri = new Uri(request.Url);
var fileName = uri.AbsolutePath;
var requestedFilePath = frontendFolderPath + fileName;
if (File.Exists(requestedFilePath))
{
byte[] bytes = File.ReadAllBytes(requestedFilePath);
Stream = new MemoryStream(bytes);
var fileExtension = Path.GetExtension(fileName);
MimeType = GetMimeType(fileExtension);
callback.Continue();
return true;
}
callback.Dispose();
return false;
}
}
public class CustomProtocolSchemeHandlerFactory : ISchemeHandlerFactory
{
public const string SchemeName = "customFileProtocol";
public IResourceHandler Create(IBrowser browser, IFrame frame, string schemeName, IRequest request)
{
return new CustomProtocolSchemeHandler();
}
}
}
And then register it before calling Cef.Initialize:
var settings = new CefSettings
{
BrowserSubprocessPath = GetCefExecutablePath()
};
settings.RegisterScheme(new CefCustomScheme
{
SchemeName = CustomProtocolSchemeHandlerFactory.SchemeName,
SchemeHandlerFactory = new CustomProtocolSchemeHandlerFactory()
});
If you simply need to return a string, then you can use ResourceHandler.FromString(html, mimeType). For this you just need to implement the ISchemeHandlerFactory.
https://github.com/cefsharp/CefSharp/blob/cefsharp/47/CefSharp/ResourceHandler.cs#L98
Example reading from a file https://github.com/cefsharp/CefSharp/blob/cefsharp/47/CefSharp.Example/CefSharpSchemeHandlerFactory.cs#L17 which can be translated to reading from a string quite simply.

HtmlAgilityPack - loading multiple pages

I've been playing around with the HtmlAgilityPack for a while, but I've run into a problem regarding the creation of a new HtmlDocument. I have a simple program that gets the data of films on a particular list. Some of the information is retrieved on the list page itself, and the rest is retrieved on the linking page for each item.
The problem i'm having is that for every time I wish to retrieve information from the linked page, i'm creating a new HtmlDocument. When I try to retrieve the complete list of films, the program just hangs on the console window. Here is my code:
namespace ConsoleApplication5
{
public class Scraper
{
private string _baseUrl = #"http://www.imdb.com";
private string _startingUrl = #"http://www.imdb.com/chart/top";
private HtmlWeb _webGet = new HtmlWeb();
public string StartingUrl
{
get { return _startingUrl; }
}
public string BaseUrl
{
get { return _baseUrl; }
}
public HtmlWeb WebGet
{
get { return _webGet; }
}
public List<Film> GetFilmData()
{
var allFilmData = new List<Film>();
var doc = WebGet.Load(StartingUrl);
var allFilmsInTable = doc.DocumentNode.SelectNodes("//div[#id='main']/table/tr");
foreach (var line in allFilmsInTable)
{
if (line.PreviousSibling != null)
{
var film = new Film();
film.Title = line.SelectSingleNode(".//td/font/a").InnerHtml;
film.Url = BaseUrl + line.SelectSingleNode(".//td/font/a").Attributes["href"].Value;
film.Rating = Convert.ToDecimal(line.SelectSingleNode(".//td[#align='center']/font").InnerText);
film.RankInTop250 = Convert.ToInt32(line.SelectSingleNode(".//td[#align='right']/font/b").InnerText.Replace(".",string.Empty));
allFilmData.Add(SingleFilmInformation(film));
}
}
return allFilmData;
}
public Film SingleFilmInformation(Film film)
{
var singleDoc = WebGet.Load(film.Url);
film.ReleaseYear = Convert.ToInt32(singleDoc.DocumentNode.SelectSingleNode("//h1[#class='header']/span/a").InnerText);
film.Director = singleDoc.DocumentNode.SelectSingleNode("//div[#itemprop='director']/a/span").InnerText;
foreach (var genre in singleDoc.DocumentNode.SelectNodes("//div[#class='infobar']/a/span[#itemprop='genre']"))
{
film.Genres.Add(genre.InnerText);
}
return film;
}
}
Any help would be greatly appreciated.

i need so input with xml and webclient in wp7

Is there anyway to read the specific line?
http://i.stack.imgur.com/hDPIg.jpg
XDocument dataFeed = XDocument.Parse(e.Result);
AchivementsListBox.ItemsSource = from query in dataFeed.Descendants("MaxPayne3")
select new NewGamesClass
{
GameGuide = (string)query.Element("Guide")
};
I would use a single WebClient instance to download the files, and simply reduce it to a single event handler. More than that, you have a similar document structure from what I can tell. That means that you could also re-use your reading code instead of writing it twice.
Something like this:
void Download(GameType type, Action onCompletion)
{
WebClient client = new WebClient();
client.DownloadProgressChanged += (s, args) =>
{
// Handle change in progress.
};
client.DownloadStringCompleted += (s, args) =>
{
XDocument feed = XDocument.Parse(args.Result);
var itemSource = from query in feed.Root
select new NewGamesClass
{
NewGameTitle = (string)query.Element("Title"),
NewGameDescription = (string)query.Element("Descript
NewGameImage = (string)query.Element("Image")
};
switch (type)
{
case GameType.ComingSoon:
{
ComingSoonList.ItemSource = itemSource;
}
case GameType.NewGames:
{
NewGameList.ItemSource = itemSource;
}
}
if (onCompletion != null)
onCompletion();
};
switch (type)
{
case GameType.NewGames:
{
client.DownloadStringAsync(new Uri("http://microsoft.com", UriKind.Absolute));
break;
}
case GameType.ComingSoon:
{
client.DownloadStringAsync(new Uri("http://www.bing.com", UriKind.Absolute));
break;
}
}
}
Although the code might look a bit more complex, it lets you recursively download data when a specific data set is downloaded. Obviously, you would have to declare the GameType enum and I simply used a bunch of test values here to demonstrate the idea.
// It Will Download The XML Once To Use further to Avoid Again And Again Calls For Each Time
public void GetXML(string path)
{
WebClient wcXML = new WebClient();
wcXML.OpenReadAsync(new Uri(path));
wcXML.OpenReadCompleted += new OpenReadCompletedEventHandler(webClient);
}
void webClient(object sender, OpenReadCompletedEventArgs e)
{
if (e.Error == null)
{
try
{
Stream Resultstream = e.Result;
XmlReader reader = XmlReader.Create(Resultstream);
var isolatedfile = IsolatedStorageFile.GetUserStoreForApplication();
using (IsolatedStorageFileStream stream = new IsolatedStorageFileStream("Download.xml", System.IO.FileMode.Create, isolatedfile))
{
byte[] buffer = new byte[e.Result.Length];
while (e.Result.Read(buffer, 0, buffer.Length) > 0)
{
stream.Write(buffer, 0, buffer.Length);
}
stream.Flush();
System.Threading.Thread.Sleep(0);
}
}
catch (Exception ex)
{
//Log Exception
}
}
if (e.Error != null)
{
//Log Exception
}
}
// This Method Will Give You Required Info According To Your Tag Like "Achievement123"
protected List<DownloadInfo> GetDetailFromXML(string TagName)
{
//TagName Like "achivement23"
List<DownloadInfo> listDetails = new List<DownloadInfo>();
XDocument loadedData;
try
{
using (var storage = IsolatedStorageFile.GetUserStoreForApplication())
{
// Check For Islated Storage And Load From That
if (storage.FileExists("Download.xml"))
{
using (Stream stream = storage.OpenFile("Download.xml", FileMode.Open, FileAccess.Read))
{
loadedData = XDocument.Load(stream);
//TagName Like "achivement23"
listDetails.AddRange((from query in loadedData.Element("GOW1").Elements(TagName)
select new DownloadInfo
{
TITLE = (string)query.Element("TITLE"),
DESCRIPTION = (string)query.Element("DESCRIPTION"),
ACHIVEMENTIMAGE = (string)query.Element("ACHIVEMENTIMAGE"),
GUIDE = (string)query.Element("GUIDE"),
YOUTUBELINK = (string)query.Element("YOUTUBELINK")
}).ToList());
}
}
}
return listDetails;
}
catch (Exception ex)
{
return listDetails = null;
//Log Exception
}
}
public class DownloadInfo
{
public string TITLE { get; set; }
public string DESCRIPTION { get; set; }
public string GUIDE { get; set; }
public string ACHIVEMENTIMAGE { get; set; }
public string YOUTUBELINK { get; set; }
}
Here is Your Method Calls
GetXML("ANY URL FROM WHERE YOU WANT TO GET XML"); // You Will Download All The XML Once To Avoid Again And Again Server Calls To Get XML.
GetDetailFromXML("YOUR TAG NAME"); // It Will Take Your Tag Name Which Information You Want To Get Like "Acheicement123" and It Will Return You Data In List<DownloadInfo> and You can easily get data from this List.
I hope it will help you. I didnot Test The Code on Runtime. But I Hope It Will Give You Some Idea.. :)

Categories

Resources