Abot Crawler Omit CrawledPage HttpWebRequest/Response - c#

I am using Abot in a way that I have a WPF application which displays a browser control (CefSharp).
The user logs in and whichever possible custom authentication the site is using will work while crawling in the same way as if the user were actually browsing the site.
Thus, when I crawl, I want to use this browser control to make the request and simply return the page data.
Therefore I've implemented my custom PageRequester, complete listing below.
The problem is that with CefSharp, as with other browser controls, it's not possible to get the HttpWebRequest/Response associated with a CrawlPage.
Without setting these two properties, Abot does not proceed the crawl further.
Is there something I can do to circumvent this problem?
Code listing:
using Abot.Core;
using Abot.Poco;
using CefSharp.Wpf;
using System;
using System.Net;
using System.Text;
using System.Threading;
public class CefPageRequester : IPageRequester
{
private MainWindowDataContext DataContext;
private ChromiumWebBrowser ChromiumWebBrowser;
private CrawlConfiguration CrawlConfig;
private volatile bool _navigationCompleted;
private string _pageSource;
public CefPageRequester(MainWindowDataContext dataContext, ChromiumWebBrowser chromiumWebBrowser, CrawlConfiguration crawlConfig)
{
this.DataContext = dataContext;
this.ChromiumWebBrowser = chromiumWebBrowser;
this.CrawlConfig = crawlConfig;
this.ChromiumWebBrowser.FrameLoadEnd += ChromiumWebBrowser_FrameLoadEnd;
}
public CrawledPage MakeRequest(Uri uri)
{
return this.MakeRequest(uri, cp => new CrawlDecision() { Allow = true });
}
public CrawledPage MakeRequest(Uri uri, Func<CrawledPage, CrawlDecision> shouldDownloadContent)
{
if (uri == null)
throw new ArgumentNullException("uri");
CrawledPage crawledPage = new CrawledPage(uri);
try
{
//the browser control is bound to the address of the data context,
//if we set the address directly it breaks for some reason, although it's a two way binding.
this.DataContext.Address = uri.AbsolutePath;
crawledPage.RequestStarted = DateTime.Now;
crawledPage.DownloadContentStarted = crawledPage.RequestStarted;
while (!_navigationCompleted)
Thread.CurrentThread.Join(10);
}
catch (WebException e)
{
crawledPage.WebException = e;
}
catch
{
//bad luck, we should log this.
}
finally
{
//TODO must add these properties!!
//crawledPage.HttpWebRequest = request;
//crawledPage.HttpWebResponse = response;
crawledPage.RequestCompleted = DateTime.Now;
crawledPage.DownloadContentCompleted = crawledPage.RequestCompleted;
if (!String.IsNullOrWhiteSpace(_pageSource))
crawledPage.Content = this.GetContent("UTF-8", _pageSource);
_navigationCompleted = false;
_pageSource = null;
}
return crawledPage;
}
private void ChromiumWebBrowser_FrameLoadEnd(object sender, CefSharp.FrameLoadEndEventArgs e)
{
if (!e.IsMainFrame)
return;
this.ChromiumWebBrowser.Dispatcher.BeginInvoke(
(Action)(() =>
{
_pageSource = this.ChromiumWebBrowser.GetSourceAsync().Result;
_navigationCompleted = true;
}));
}
private PageContent GetContent(string charset, string html)
{
PageContent pageContent = new PageContent();
pageContent.Charset = charset;
pageContent.Encoding = this.GetEncoding(charset);
pageContent.Text = html;
pageContent.Bytes = pageContent.Encoding.GetBytes(html);
return pageContent;
}
private Encoding GetEncoding(string charset)
{
Encoding e = Encoding.UTF8;
if (charset != null)
{
try
{
e = Encoding.GetEncoding(charset);
}
catch { }
}
return e;
}
}
The question can also be phrased as: how to avoid having to create a HttpWebResponse from a stream? Which seems impossible, given MSDN says:
You should never directly create an instance of the HttpWebResponse
class. Instead, use the instance returned by a call to
HttpWebRequest.GetResponse.
I would have to actually post the request to get the response, which is precisely what I want to avoid by having a web browser control.

As you are aware, lots of functionality depends on the HttpWebRequest and HttpWebResponse being set. I've ordered a few options for you off the top of my head...
1) Refactor Abot to use some POCO Abstraction instead of those classes. Then just have an converter that converts the real HttpWebRequest and HttpWebResponse to those POCO types as well as a converter that converts your browser objects response into those POCOs.
2) Create a CustomHttpWebRequest and CustomHttpWebResponse that inherit from the .net classes so you can access/override the public/protected properties which may allow you to manually create an instance that models the request/response that your browser component returns to you. I know this can be tricky but may work (I've never done it so I can't say for sure).
3) [I HATE THIS IDEA. It SHOULD BE YOUR LAST RESORT] Create a real instance of these classes and use reflection to set whatever properties/values need to be set to satisfy all of Abot's usages.
4) [I HATE THIS IDEA EVEN WORSE] Use MS Fakes to create shims/stubs/fakes to the properties and methods of the HttpWebRequest and HttpWebResponse. Then you could configure it to return your values. This tool is usually only used for testing but I believe it can be used for production code if you are desperate, don't care about performance and/or are insane.
I also included the terrible ideas as well to just in case they help you spark some thought. Hope that helps...

Related

How to send data as a SoapMessage and get a reply?

I have some data that needs to be send in SOAP format to a server. This server will immediately acknowledge that it received the messages. After a few hours I get (possibly from another server) a SOAP message that contains information about the processed data.
I read Stackoverflow: How to send SOAP request and receive response. However, the answers are 8 years old. Although they may still work, It may be that there are newer techniques.
And indeed it seems: Microsoft has System.Web.Services.Protocols, with classes like SoapMessage, SoapClientMessage, SoapServerMessage, etc.
Looking at the classes I find a lot of SOAP like classes (headers, extensions, client messages, server messages... Normally the provided examples give me an indication to how these classes work together and how to use them. In the MSDN documents I can only find examples of how to process already existing SOAP messages.
Given some data that needs to be sent, how can I wrap this data somehow in one of these SOAP classes and send this message?
Are these classes meant for this purpose? Or should I stick to the 2011 method where you'd create a SOAP Web request by formatting the XML data in soap format yourself, as the above mentioned Stackoverflow question suggests?
I'm awfully sorry, normally I would write things I have tried. Alas I don't see the relation between the provided SoapMessage classes. I haven't got a clue how to use them.
Addition after comments
I'm using windows server / visual studio (newest versions) / .NET (newest versions) / C# (newest versions).
The communication with the server is mutual authenticated. The certificate that I need to use to communicate with the server, is in PEM (CER / CRT) format. The privated key is RSA. This certificate is issued by a proper CA, the server will also use certificates used by a proper CA. So I don't need to create a new certificate (in fact, it won't be accepted). If needed, I'm willing to convert the certificates using programs like OpenSsl and the like.
I've tried to use Apache TomCat to communicate, but I have the feeling that that's way too much for the task of sending one SOAP message per day and waiting for one answer per day.
Maybe because java is a complete new technique for me, it was difficult for me to see the contents of the received messages. So back to C# and .NET.
I was planning to create a DLL, to be used by a console app. The function would have some data in a stream as input. It would create the soap message, send it, wait for reply that the message was received correctly, and wait (possible several hours) for a new Soap message containing the results of the processed data. To make proper reporting, and cancellation possible, I guess it is best to do this using async-await
If sending the order and waiting for the result can't be done in one application, I'm willing to create a windows service that that listens to the input, but I prefer to keep it simple.
The (virtual) computer will only be used for this task, so no one else will need to listen to port 443. There will be one order message send per day, and one result message per day.
Here is sample C# Console client and server code (they are in the same sample but this is only for demo purpose, of course) that uses HTTPS.
For the client side, we reuse the SoapHttpClientProtocol class, but for the server side, unfortunately, we cannot reuse anything because classes are completely tied to ASP.NET's (IIS) HttpContext class
For the server side, we use HttpListener, so, depending on your configuration, the server side will probably require admin rights to be able to call HttpListener's Prefixes.Add(url).
The code doesn't uses client certificate, but you can add this where I placed // TODO comments
The code assumes there is a certificate associated with the url and port used. If there's not (use netsh http show sslcert to dump all associated certs), you can use the procedure described here to add one: https://stackoverflow.com/a/11457719/403671
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using System.Web.Services;
using System.Web.Services.Protocols;
using System.Xml;
namespace SoapTests
{
class Program
{
static void Main(string[] args)
{
// code presumes there is an sslcert associated with the url/port below
var url = "https://127.0.0.1:443/";
using (var server = new MyServer(url, MyClient.NamespaceUri))
{
server.Start(); // requests will occur on other threads
using (var client = new MyClient())
{
client.Url = url;
Console.WriteLine(client.SendTextAsync("hello world").Result);
}
}
}
}
[WebServiceBinding(Namespace = NamespaceUri)]
public class MyClient : SoapHttpClientProtocol
{
public const string NamespaceUri = "http://myclient.org/";
public async Task<string> SendTextAsync(string text)
{
// TODO: add client certificates using this.ClientCertificates property
var result = await InvokeAsync(nameof(SendText), new object[] { text }).ConfigureAwait(false);
return result?[0]?.ToString();
}
// using this method is not recommended, as async is preferred
// but we need it with this attribute to make underlying implementation happy
[SoapDocumentMethod]
public string SendText(string text) => SendTextAsync(text).Result;
// this is the new Task-based async model (TAP) wrapping the old Async programming model (APM)
public Task<object[]> InvokeAsync(string methodName, object[] input, object state = null)
{
if (methodName == null)
throw new ArgumentNullException(nameof(methodName));
return Task<object[]>.Factory.FromAsync(
beginMethod: (i, c, o) => BeginInvoke(methodName, i, c, o),
endMethod: EndInvoke,
arg1: input,
state: state);
}
}
// server implementation
public class MyServer : TinySoapServer
{
public MyServer(string url, string namespaceUri)
: base(url)
{
if (namespaceUri == null)
throw new ArgumentNullException(nameof(namespaceUri));
NamespaceUri = namespaceUri;
}
// must be same as client namespace in attribute
public override string NamespaceUri { get; }
protected override bool HandleSoapMethod(XmlDocument outputDocument, XmlElement requestMethodElement, XmlElement responseMethodElement)
{
switch (requestMethodElement.LocalName)
{
case "SendText":
// get the input
var text = requestMethodElement["text", NamespaceUri]?.InnerText;
text += " from server";
AddSoapResult(outputDocument, requestMethodElement, responseMethodElement, text);
return true;
}
return false;
}
}
// simple generic SOAP server
public abstract class TinySoapServer : IDisposable
{
private readonly HttpListener _listener;
protected TinySoapServer(string url)
{
if (url == null)
throw new ArgumentNullException(nameof(url));
_listener = new HttpListener();
_listener.Prefixes.Add(url); // this requires some rights if not used on localhost
}
public abstract string NamespaceUri { get; }
protected abstract bool HandleSoapMethod(XmlDocument outputDocument, XmlElement requestMethodElement, XmlElement responseMethodElement);
public async void Start()
{
_listener.Start();
do
{
var ctx = await _listener.GetContextAsync().ConfigureAwait(false);
ProcessRequest(ctx);
}
while (true);
}
protected virtual void ProcessRequest(HttpListenerContext context)
{
if (context == null)
throw new ArgumentNullException(nameof(context));
// TODO: add a call to context.Request.GetClientCertificate() to validate client cert
using (var stream = context.Response.OutputStream)
{
ProcessSoapRequest(context, stream);
}
}
protected virtual void AddSoapResult(XmlDocument outputDocument, XmlElement requestMethodElement, XmlElement responseMethodElement, string innerText)
{
if (outputDocument == null)
throw new ArgumentNullException(nameof(outputDocument));
if (requestMethodElement == null)
throw new ArgumentNullException(nameof(requestMethodElement));
if (responseMethodElement == null)
throw new ArgumentNullException(nameof(responseMethodElement));
var result = outputDocument.CreateElement(requestMethodElement.LocalName + "Result", NamespaceUri);
responseMethodElement.AppendChild(result);
result.InnerText = innerText ?? string.Empty;
}
protected virtual void ProcessSoapRequest(HttpListenerContext context, Stream outputStream)
{
// parse input
var input = new XmlDocument();
input.Load(context.Request.InputStream);
var ns = new XmlNamespaceManager(new NameTable());
const string soapNsUri = "http://schemas.xmlsoap.org/soap/envelope/";
ns.AddNamespace("soap", soapNsUri);
ns.AddNamespace("x", NamespaceUri);
// prepare output
var output = new XmlDocument();
output.LoadXml("<Envelope xmlns='" + soapNsUri + "'><Body/></Envelope>");
var body = output.SelectSingleNode("//soap:Body", ns);
// get the method name, select the first node in our custom namespace
bool handled = false;
if (input.SelectSingleNode("//x:*", ns) is XmlElement requestElement)
{
var responseElement = output.CreateElement(requestElement.LocalName + "Response", NamespaceUri);
body.AppendChild(responseElement);
if (HandleSoapMethod(output, requestElement, responseElement))
{
context.Response.ContentType = "application/soap+xml; charset=utf-8";
context.Response.StatusCode = (int)HttpStatusCode.OK;
var writer = new XmlTextWriter(outputStream, Encoding.UTF8);
output.WriteTo(writer);
writer.Flush();
handled = true;
}
}
if (!handled)
{
context.Response.StatusCode = (int)HttpStatusCode.BadRequest;
}
}
public void Stop() => _listener.Stop();
public virtual void Dispose() => _listener.Close();
}
}
Personally, I use ServiceStack to create both client and server
https://docs.servicestack.net/soap-support
Or SoapHttpClient nuget
https://github.com/pmorelli92/SoapHttpClient
Or my example from way back when
Is it possible that I can convert simple string to SOAP Message and send it?
The answer depends on what framework or libraries do you plan to use?
The simplest modern answer is to declare a simple class that defines the structure of your message and then serialize it using HttpClient to send it.
However, SOAP is a standard built for description based messaging so the still relevant recommendation is to generate your client code from the wsdl description using a "service reference" then use the generated client object.
I would however recommend, like others have pointed out that you try to move to REST services instead (assuming this is possible).
The code is less complex, the system is far simpler to use and it's a global standard.
Here is a comparison and example of both ...
https://smartbear.com/blog/test-and-monitor/understanding-soap-and-rest-basics/

Sharing Informations between Form/Class [duplicate]

This question already has an answer here:
Interaction between forms — How to change a control of a form from another form?
(1 answer)
Closed 5 years ago.
I know the Title Issue is not unique and lot of Messages available. Reason why I am opening a thread is that I'm really struggling to understand the issue based on my project.
Recently I tried to start a WinForm to get values from an REST API.
Just as a beginning test on the Form1 once clicking the metroButton1 I should get a SessionToken to the metroTextBox1.Text, which works.
Where I'm struggling is to use that Output as an Input for the getvirtualmachine Class.
Once using the Form1.connect() in the class to get the SessionToken, i'am getting following Error:
An object reference is required for the non-static field, method, or property 'Form1.connect()'
Not sure how I could make that globally to use it as Input for several classes. Guessing I'm making somehow/somewhere a big mistake.
FORM1
public partial class Form1 : MetroFramework.Forms.MetroForm
{
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{ }
public string connect()
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://localhost:35113/api/sessions/start");
httpWebRequest.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(metroTextBox3.Text + ":" + metroTextBox4.Text));
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = JsonConvert.SerializeObject(new
{
ServerPort = "35107",
Username = metroTextBox3.Text,
Password = metroTextBox4.Text,
Domain = metroTextBox5.Text
});
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
dynamic item = Newtonsoft.Json.JsonConvert.DeserializeObject(result);
return (string)item.Data;
}
}
public void metroButton1_Click(object sender, EventArgs e)
{
metroTextBox1.Text = connect();
}
GetVirtualMachines Class
public class GetVirtualMachines
{
public string gVM()
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create("http://localhost:35113/api/vms/list/");
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "GET";
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = JsonConvert.SerializeObject(new
{
SessionToken = Form1.connect(),
});
streamWriter.Write(json);
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
dynamic item = Newtonsoft.Json.JsonConvert.DeserializeObject(result);
return item;
}
}
}
You either need to make the method static, so it's part of the class definition rather than an instance method, i.e.
public static string connect()
or create an instance of Form1 and use that to call connect(). However, that ends up creating another problem because now you've got a form hanging out in memory that no one ever sees (and will prevent the application from closing), so you really should ensure it gets disposed, i.e.
using(var form = new Form1)
{
form.connect();
}
However, my bigger question would be, why are you using a Form to perform this operation at all. It seems to me more like the connect() method belongs on the GetVirtualMachines class.
At the moment, given the code you've shown, this is really bad use of the toolset.
You need to rethink your approach here. The Windows Form (Form1) is the actual GUI that the user sees. Your backend GetVMs class appears to be something purely for data processing that the user doesn't really interact with.
So you can't really have the GetVMs class create a new instance of your Windows Form because, well, there is no way for the user to see the form that you new up (simplifying a lot here!).
I would solve this by following a relatively standard view/controller type pattern:
Your windows form is the view -- that is what the user sees
The form has an instance of your GetVMs class private GetVirtualMachines VMGetter
The form calls methods of GetVirtualMachines via its instance of the class (VMGetter), passing an argument with whatever is in the text box.
This is still not the greatest idea in the simplified approach I gave because you'll hang the UI thread. But, hopefully it gives you a jumping off point.
Edit: I'd also move the connect method and its business logic to your GetVMs class. Typically we try to avoid business logic in all view/display classes.

Xilium.CefGlue how to execute JavaScript with return Value?

I'm quite new in programming multi-threading and I could not understand from the xelium example how I could execute a javascript and get the return value.
I have tested:
browser.GetMainFrame().ExecuteJavaScript("SetContent('my Text.')", null, 0);
the javascript is executed, but I this function don’t allow me to get the return value.
I should execute the following function to get all the text the user have written in the box..
browser.GetMainFrame().ExecuteJavaScript("getContent('')", null, 0);
the function TryEval should do this…
browser.GetMainFrame().V8Context.TryEval("GetDirtyFlag", out returninformation , out exx);
But this function can’t be called from the browser, I think it must be called from the renderer? How can I do so?
I couldn’t understand the explanations about CefRenderProcessHandler and OnProcessMessageReceived.. How to register a Scriptable Object and set my javascript & parameters?
Thx for any suggestions how I could solve this!
I have been struggling with this as well. I do not think there is a way to do this synchronously...or easily :)
Perhaps what can be done is this:
From browser do sendProcessMessage with all JS information to renderer
process. You can pass all kinds of parameters to this call in a structured way so encapsulating the JS method name and params in order should not be difficult to do.
In renderer process (RenderProcessHandler onProcessMessageReceived method) do TryEval on the V8Context and get the return value via out parameters and sendProcessMessage back to the
browser process with the JS return value (Note that this supports ordinary return semantics from your JS method).You get the browser instance reference in the onProcessMessageReceived so it is as easy as this (mixed pseudo code)
browser.GetMainFrame().CefV8Context.tryEval(js-code,out retValue, out exception);
process retValue;
browser.sendProcessMessage(...);
Browser will get a callback in the WebClient in onProcessMessageReceived.
There is nothing special here in terms of setting up JS. I have for example a loaded html page with a js function in it. It takes a param as input and returns a string. in js-code parameter to TryEval I simply provide this value:
"myJSFunctionName('here I am - input param')"
It is slightly convoluted but seems like a neat workable approach - better than doing ExecuteJavaScript and posting results via XHR on custom handler in my view.
I tried this and it does work quite well indeed....and is not bad as it is all non-blocking. The wiring in the browser process needs to be done to process the response properly.
This can be extended and built into a set of classes to abstract this out for all kinds of calls..
Take a look at the Xilium demo app. Most of the necessary wiring is already there for onProcessMessage - do a global search. Look for
DemoRendererProcessHandler.cs - renderer side this is where you will invoke tryEval
DemoApp.cs - this is browser side, look for sendProcessMessage - this will initiate your JS invocation process.
WebClient.cs - this is browser side. Here you receive messages from renderer with return value from your JS
Cheers.
I resolved this problem by returning the result value from my JavaScript function back to Xilium host application via an ajax call to a custom scheme handler. According to Xilium's author fddima it is the easiest way to do IPC.
You can find an example of how to implement a scheme handler in the Xilium's demo app.
Check out this post: https://groups.google.com/forum/#!topic/cefglue/CziVAo8Ojg4
using System;
using System.Windows.Forms;
using Xilium.CefGlue;
using Xilium.CefGlue.WindowsForms;
namespace CefGlue3
{
public partial class Form1 : Form
{
private CefWebBrowser browser;
public Form1()
{
InitializeCef();
InitializeComponent();
}
private static void InitializeCef()
{
CefRuntime.Load();
CefMainArgs cefArgs = new CefMainArgs(new[] {"--force-renderer-accessibility"});
CefApplication cefApp = new CefApplication();
CefRuntime.ExecuteProcess(cefArgs, cefApp);
CefSettings cefSettings = new CefSettings
{
SingleProcess = false,
MultiThreadedMessageLoop = true,
LogSeverity = CefLogSeverity.ErrorReport,
LogFile = "CefGlue.log",
};
CefRuntime.Initialize(cefArgs, cefSettings, cefApp);
}
private void Form1_Load(object sender, EventArgs e)
{
browser = new CefWebBrowser
{
Visible = true,
//StartUrl = "http://www.google.com",
Dock = DockStyle.Fill,
Parent = this
};
Controls.Add(browser);
browser.BrowserCreated += BrowserOnBrowserCreated;
}
private void BrowserOnBrowserCreated(object sender, EventArgs eventArgs)
{
browser.Browser.GetMainFrame().LoadUrl("http://www.google.com");
}
}
}
using Xilium.CefGlue;
namespace CefGlue3
{
internal sealed class CefApplication : CefApp
{
protected override CefRenderProcessHandler GetRenderProcessHandler()
{
return new RenderProcessHandler();
}
}
internal sealed class RenderProcessHandler : CefRenderProcessHandler
{
protected override void OnWebKitInitialized()
{
CefRuntime.RegisterExtension("testExtension", "var test;if (!test)test = {};(function() {test.myval = 'My Value!';})();", null);
base.OnWebKitInitialized();
}
}
}

2 asynchronus download requests, but the second is dependent on the result of the first - how to sync

Edit: Sorry - now that I've understood the problem a bit better, I think my problem lies elsewhere
I have 2 asynchronus requests.
The first is this:
public void DownloadWebData(Uri apiUrl)
{
WebClient client = new WebClient();
client.DownloadDataCompleted += DownloadDataCompleted;
client.DownloadDataAsync(apiUrl);
}
public void DownloadDataCompleted(object sender, DownloadDataCompletedEventArgs e)
{
string result = System.Text.Encoding.UTF8.GetString (e.Result);
Uri downloadLink = (GetUri(result));
}
Basically it makes a simple url based API request to a remote webserver which returns some basic textual data over http. GetUri() just parses that data to extract an address from the data for an image to download.
I'm then using imageLoader in monotouch.dialog to download the image. All code is in the same class.
Edit: added the imageLoader code (I left the Console lines in because they serve reasonably well as comments).
public void downloadImage (Uri imageUri)
{
var tmp_img = ImageLoader.DefaultRequestImage (imageUri, this);
if (tmp_img != null)
{
adView.Image = tmp_img;
Console.WriteLine ("Image already cached, displaying");
}
else
{
adView.Image = UIImage.FromFile ("Images/downloading.jpg");
Console.WriteLine ("Image not cached. Using placeholder.");
}
}
public void UpdatedImage (System.Uri uri)
{
adView.Image = ImageLoader.DefaultRequestImage(uri, this);
}
You missed to check if e.Result actually contains something. The download might as well have failed and e.Result is null. Add some basic error handling to your code.
if you are using DownloadWebData inside a for loop, it will be better you generate seperate functions for DownloadDataCompleted event.
You can use anonymous function inside DownloadWebData().
client.DownloadDataCompleted +=(s,e)=>{
string result = System.Text.Encoding.UTF8.GetString (e.Result);
Uri downloadLink = (GetUri(result));
};
After realizing I was asking the wrong question, I finally figured it out here:
Hand back control to main UI thread to update UI after asynchronus image download

Store the cache data locally

I develops a C# Winform application, it is a client and connect to web service to get data. The data returned by webservice is a DataTable. Client will display it on a DataGridView.
My problem is that: Client will take more time to get all data from server (web service is not local with client). So I must to use a thread to get data. This is my model:
Client create a thread to get data -> thread complete and send event to client -> client display data on datagridview on a form.
However, when user closes the form, user can open this form in another time, and client must get data again. This solution will cause the client slowly.
So, I think about a cached data:
Client <---get/add/edit/delete---> Cached Data ---get/add/edit/delete--->Server (web service)
Please give me some suggestions.
Example: cached data should be developed in another application which is same host with client? Or cached data is running in client.
Please give me some techniques to implement this solution.
If having any examples, please give me.
Thanks.
UPDATE : Hello everyone, maybe you think my problem so far. I only want to cache data in client's lifetime. I think cache data should be stored in memory. And when client want to get data, it will check from cache.
If you're using C# 2.0 and you're prepared to ship System.Web as a dependency, then you can use the ASP.NET cache:
using System.Web;
using System.Web.Caching;
Cache webCache;
webCache = HttpContext.Current.Cache;
// See if there's a cached item already
cachedObject = webCache.Get("MyCacheItem");
if (cachedObject == null)
{
// If there's nothing in the cache, call the web service to get a new item
webServiceResult = new Object();
// Cache the web service result for five minutes
webCache.Add("MyCacheItem", webServiceResult, null, DateTime.Now.AddMinutes(5), Cache.NoSlidingExpiration, System.Web.Caching.CacheItemPriority.Normal, null);
}
else
{
// Item already in the cache - cast it to the right type
webServiceResult = (object)cachedObject;
}
If you're not prepared to ship System.Web, then you might want to take a look at the Enterprise Library Caching block.
If you're on .NET 4.0, however, caching has been pushed into the System.Runtime.Caching namespace. To use this, you'll need to add a reference to System.Runtime.Caching, and then your code will look something like this:
using System.Runtime.Caching;
MemoryCache cache;
object cachedObject;
object webServiceResult;
cache = new MemoryCache("StackOverflow");
cachedObject = cache.Get("MyCacheItem");
if (cachedObject == null)
{
// Call the web service
webServiceResult = new Object();
cache.Add("MyCacheItem", webServiceResult, DateTime.Now.AddMinutes(5));
}
else
{
webServiceResult = (object)cachedObject;
}
All these caches run in-process to the client. Because your data is coming from a web service, as Adam says, you're going to have difficulty determining the freshness of the data - you'll have to make a judgement call on how often the data changes and how long you cache the data for.
Do you have the ability to make changes/add to the webservice?
If you can Sync Services may be an option for you. You can define which tables are syncronised, and all the sync stuff is managed for you.
Check out
http://msdn.microsoft.com/en-us/sync/default.aspx
and shout if you need more information.
You might try the Enterprise Library's Caching Application Block. It's easy to use, stores in memory and, if you ever need to later, it supports adding a backup location for persisting beyond the life of the application (such as to a database, isolated storage, file, etc.) and even encryption too.
Use EntLib 3.1 if you're stuck with .NET 2.0. There's not much new (for caching, at least) in the newer EntLibs aside from better customization support.
Identify which objects you would like to serialize, and cache to isolated storage. Specify the level of data isolation you would like (application level, user level, etc).
Example:
You could create a generic serializer, a very basic sample would look like this:
public class SampleDataSerializer
{
public static void Deserialize<T>(out T data, Stream stm)
{
var xs = new XmlSerializer(typeof(T));
data = (T)xs.Deserialize(stm);
}
public static void Serialize<T>(T data, Stream stm)
{
try
{
var xs = new XmlSerializer(typeof(T));
xs.Serialize(stm, data);
}
catch (Exception e)
{
throw;
}
}
}
Note that you probably should put in some overloads to the Serialize and Deserialize methods to accomodate readers, or any other types you are actually using in your app (e.g., XmlDocuments, etc).
The operation to save to IsolatedStorage can be handled by a utility class (example below):
public class SampleIsolatedStorageManager : IDisposable
{
private string filename;
private string directoryname;
IsolatedStorageFile isf;
public SampleIsolatedStorageManager()
{
filename = string.Empty;
directoryname = string.Empty;
// create an ISF scoped to domain user...
isf = IsolatedStorageFile.GetStore(IsolatedStorageScope.User |
IsolatedStorageScope.Assembly | IsolatedStorageScope.Domain,
typeof(System.Security.Policy.Url), typeof(System.Security.Policy.Url));
}
public void Save<T>(T parm)
{
using (IsolatedStorageFileStream stm = GetStreamByStoredType<T>(FileMode.Create))
{
SampleDataSerializer.Serialize<T>(parm, stm);
}
}
public T Restore<T>() where T : new()
{
try
{
if (GetFileNameByType<T>().Length > 0)
{
T result = new T();
using (IsolatedStorageFileStream stm = GetStreamByStoredType<T>(FileMode.Open))
{
SampleDataSerializer.Deserialize<T>(out result, stm);
}
return result;
}
else
{
return default(T);
}
}
catch
{
try
{
Clear<T>();
}
catch
{
}
return default(T);
}
}
public void Clear<T>()
{
if (isf.GetFileNames(GetFileNameByType<T>()).Length > 0)
{
isf.DeleteFile(GetFileNameByType<T>());
}
}
private string GetFileNameByType<T>()
{
return typeof(T).Name + ".cache";
}
private IsolatedStorageFileStream GetStreamByStoredType<T>(FileMode mode)
{
var stm = new IsolatedStorageFileStream(GetFileNameByType<T>(), mode, isf);
return stm;
}
#region IDisposable Members
public void Dispose()
{
isf.Close();
}
}
Finally, remember to add the following using clauses:
using System.IO;
using System.IO.IsolatedStorage;
using System.Xml.Serialization;
The actual code to use the classes above could look like this:
var myClass = new MyClass();
myClass.name = "something";
using (var mgr = new SampleIsolatedStorageManager())
{
mgr.Save<MyClass>(myClass);
}
This will save the instance you specify to be saved to the isolated storage. To retrieve the instance, simply call:
using (var mgr = new SampleIsolatedStorageManager())
{
mgr.Restore<MyClass>();
}
Note: the sample I've provided only supports one serialized instance per type. I'm not sure if you need more than that. Make whatever modifications you need to support further functionalities.
HTH!
You can serialise the DataTable to file:
http://forums.asp.net/t/1441971.aspx
Your only concern then is deciding when the cache has gone stale. Perhaps timestamp the file?
In our implementation every row in the database has a last-updated timestamp. Every time our client application accesses a table we select the latest last-updated timestamp from the cache and send that value to the server. The server responds with all the rows that have newer timestamps.

Categories

Resources