I asked this question yesterday.
Essentially, I'm trying to parse an XML from a URL but my code hangs forever if the connection is lost when attempting to read the XML.
I am still having the same problem, however I changed the code in a way I thought would prevent the program from freezing if the connection to the URL was interrupted. Could someone please explain why my solution didn't work and how I can fix it? Thanks!
Here are the two functions I am using. CanReach just checks the connection to make sure the URL is there, and GetTags gets all the parent tags of the XML file. I want it to break if the connection is interrupted. I tried to do this by loading the xml file instead of parsing it right from the URL and using try and catch to catch the error. xmlLocation is the URL.
public static bool CanReach(string xmlLocation)
{
WebRequest request = WebRequest.Create(xmlLocation);
request.Timeout = 1000;
try
{
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
response.Dispose();
request.Abort();
return true;
}
catch (System.Net.WebException)
{
request.Abort();
return false;
}
}
public static List<string> GetTopTags(string xmlLocation)
{
bool canBeReached = CanReach(xmlLocation);
if (canBeReached)
{
try
{
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(xmlLocation);
XmlReader reader = new XmlNodeReader(xmlDoc);
List<string> dataList = new List<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Text:
dataList.Add(reader.Name);
break;
}
}
reader.Dispose();
return topTags;
}
catch
{
return null;
}
}
else
{
return null;
}
}
We can start of another thread that is reading the XML file and the current method can continuously check whether connection is alive or not. If the we can not reach the URL anymore, we can cancel the operation and return null. If the read operation is finished, isFinished flag is set and we can return returnValue.
Try this:
public static List<string> GetTopTags(string xmlLocation)
{
bool canBeReached = CanReach(xmlLocation);
if (!canBeReached)
return null;
List<string> returnValue = null;
CancellationTokenSource cts = new CancellationTokenSource();
bool isFinished = false;
Task.Factory.StartNew(() =>
{
try
{
var xmlDoc = new XmlDocument();
xmlDoc.Load(xmlLocation);
using var reader = new XmlNodeReader(xmlDoc);
List<string> dataList = new List<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Text:
dataList.Add(reader.Name);
break;
}
}
if (reader.ReadState == ReadState.Error)
returnValue = null;
else
returnValue = topTags;
}
catch
{
returnValue = null;
}
isFinished = true;
}, cts.Token, TaskCreationOptions.LongRunning, TaskScheduler.Default);
while (!isFinished)
if (!CanReach(xmlLocation))
cts.Cancel();
return returnValue;
}
I'm not wild about that XmlDocument.Load() and passing it a URL. You're handing over too much control and it's making it hard for you to debug. I would separate out the networking and the XML reading. With the networking separated out, you eliminate the need for your CanReach() function. Anything network related needs to be ran in a separate thread. Based on your source, something like the following is what I might start with.
public static async Task<List<string>> GetTopTags(string xmlLocation)
{
string xmlText = null;
try
{
// You are free to use WebRequest here, I've used WebClient for simplicity.
using (var webClient = new WebClient())
{
xmlText = await webClient.DownloadStringTaskAsync(xmlLocation);
}
}
catch (Exception)
{
// Handle network related issues.
}
if (string.IsNullOrWhiteSpace(xmlText))
{
// We weren't able to download the XML, or the downloaded XML is not valid,
// "CanReach()" is false.
return null;
}
// We downloaded the XML successfully if you get here, now just read it.
XmlDocument xmlDoc = new XmlDocument();
xmlDoc.Load(new StringReader(xmlText));
using (XmlReader reader = new XmlNodeReader(xmlDoc))
{
List<string> dataList = new List<string>();
while (reader.Read())
{
switch (reader.NodeType)
{
case XmlNodeType.Text:
dataList.Add(reader.Name);
break;
}
}
return dataList;
}
}
Related
I have code that basically opens a webpage + .ts file from a link and repeats it, but the problem is it increases memory usage each time and never removes the old data. After 2 Hours it uses more than 2GB.
Any ideas on how I can fix this issue?
I'm using "Leaf.Xnet" Library for requests and this is how I create my threads:
new Thread(new ThreadStart(WebHelper.Check)).Start();
Main code:
public static void Check()
{
HttpRequest request = null;
while (Form1.isRuning)
{
Application.DoEvents();
try
{
request = new HttpRequest();
if (!ProxyManager.updating)
{
switch (ProxyManager.curProxyType)
{
case ProxyManager.proxyType.http:
request.Proxy = HttpProxyClient.Parse(ProxyManager.NextProxy(ProxyManager.proxyType.http));
break;
case ProxyManager.proxyType.socks4:
request.Proxy = Socks4ProxyClient.Parse(ProxyManager.NextProxy(ProxyManager.proxyType.socks4));
break;
case ProxyManager.proxyType.socks5:
request.Proxy = Socks5ProxyClient.Parse(ProxyManager.NextProxy(ProxyManager.proxyType.socks5));
break;
}
}
else
{
Thread.Sleep(2000);
Check();
}
request.UserAgentRandomize();
request.AddHeader(HttpHeader.Referer, "https://somesite.com");
request.KeepAlive = true;
request.ConnectTimeout = Form1.timeOut;
request.Reconnect = true;
string html = request.Get(Form1.link, null).ToString();
string auth = html.Substring(",[{\"src\":\"", "\"");
string sign = html.Substring("144p.apt?wmsAuthSign=", "\"");
if (auth != null && sign != null)
{
string auth2 = "";
foreach (char item in auth)
{
if (item != '\\')
auth2 += item;
}
auth = auth2;
string cdn = auth.Substring("https://", ".");
string id = auth.Substring("video/", "-");
if (cdn != null && id != null)
{
Random rnd = new Random();
request.Get(auth);
Form1.sended++;
WriteStat();
}
html = null;
auth = null;
auth2 = null;
sign = null;
}
}
catch (HttpException)
{
Check();
}
catch (ProxyException)
{
Check();
}
}
}
I am not entirely sure if this will fix your problem but for each thread that you start, you pretty much call an infinite number of executions of Check(). Since Check contains a while loop, the thread will run whatever is in side forever anyway, and now you're calling the method again on top of it. This means that everything that was created in the scope of the Check method will not be garbage collected and will increase your memory.
Replace all calls to Check() with continue which will stop the execution in the while loop and start over.
Also, consider not using Threads, but instead use Tasks.
Also you do not dispose your HttpRequest.
I have two reliable queues and they are being accessed by two guest executables and each of them access their own. Sometimes the function I use to access them doesn't update the reliable queue object in the function and the wrong request is sent to the wrong guest executable.
What happens is that the clientId is passed by the guest executable to this function in the Get request. Let us say that there are two clientId(s) called T1 and T2.
What happens is that the guest executable (client) T2 at times gets the request that was meant for T1. Even though I tried line by line debugging the parameters passed to this function are correct.
Here is my API's POST that is passed a json to be added to the queue for the clients to receive from the GET
[HttpPost("MarketInfo")]
public JObject GetMarketInfo([FromBody] JObject jObject)
{
List<JToken> clients = jObject.GetValue("clients").ToList();
string json;
JObject response = new JObject();
JArray jsonArray = new JArray();
try
{
foreach (JToken client in clients)
{
var id = Guid.NewGuid();
json = "{'name':'MarketInfo','id':'" + id.ToString() + "','mtClientId':'" + terminal["name"].ToString() + "','parameters':{'symbol':'" + terminal["symbol"].ToString() + "','property':24}}";
bool result = _requestsCollectionHandler.CreateRequestForClient(JObject.Parse(json));
JObject clientResponse = new JObject();
if (result==true)
{
clientResponse["name"] = client["name"].ToString();
clientResponse["guid"] = id.ToString();
jsonArray.Add(clientResponse);
}
else
{
clientResponse["name"] = terminal.Children()["name"].ToString();
clientResponse["guid"] = "ERROR";
jsonArray.Add(terminalResponse);
}
}
response["clients"] = jsonArray;
return response;
}
catch (Exception e)
{
Debug.Write(e.Message);
return null;
}
}
This is the json that we pass to this API
{"clients":[{"name":"T1","symbol":"SomeInfo"},{"name":"T2","symbol":"SomeInfo"}]}
The problem is always with the clients object that is passed first.
Before I explain further let me also share the code for the client's HttpGet
[HttpGet("{clientId}")]
public string Get([FromRoute] string clientId)
{
try
{
string request = _requestsCollectionHandler.GetRequestJsonFromQueue(clientId);
return request;
}
catch(Exception e)
{
return e.Message;
}
}
This is the function that creates an object that is to be added by another function in the reliable queue
public bool CreateRequestForClient(JObject jObject)
{
try
{
this._jObject = new JObject(jObject);
CreateKey();
AddToRequestToQueueAsync();
return true;
}
catch (Exception e)
{
Debug.Write(e.Message);
_exceptionMessage = e.Message;
return false;
}
}
private void CreateKey()
{
dynamic data = JObject.Parse(_jObject.ToString(Newtonsoft.Json.Formatting.None));
string name = data.name;
string id = data.id;
string clientId = data.clientId;
_key.id = id;
_key.name = name;
_key.clientId = clientId;
//key.timestamp = GetTimestamp();
_key.timestamp = GetTimestamp();
_key.requestJson = _jObject.ToString(Newtonsoft.Json.Formatting.None);
}
_key is a private variable in class a custom class
This is the function in my class of request handler that adds the requests to the queue
private void AddToRequestToQueueAsync()
{
var transaction = this._stateManager.CreateTransaction();
CancellationToken cancellationToken
= new CancellationToken(false);
try
{
string queue = _key.clientId;
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
_stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
transaction = this._stateManager.CreateTransaction();
if (reliableQueue!=null)
{
long count = reliableQueue.Count;
reliableQueue.EnqueueAsync(transaction, _key);
count = reliableQueue.Count;
transaction.CommitAsync().Wait();
}
else
{
transaction.Abort();
}
}
catch
{
transaction.Abort();
throw;
}
}
This is function that is used by the client
public string GetRequestJsonFromQueue(string clientId)
{
string queue = clientId;
try
{
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
if(reliableQueue != null)
{
ConditionalValue<TerminalResponseKey> key =
reliableQueue.TryDequeueAsync(transaction).Result;
if(key.HasValue)
{
string request = key.Value.requestJson;
transaction.CommitAsync().Wait();
return request;
}
}
else
{
transaction.Abort();
}
return "NO QUEUE";
}
catch (Exception e)
{
Debug.WriteLine(e);
transaction.Abort();
return e.InnerException.Message;
}
}
As far as I have found out I think my problem is in this function above. Because I don't know how the client T2 or client T1 gets another client's queue because the parameters determining the queue are their IDs and are totally unique.
These Ids are also passed correctly to this:
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
As you can see that we have queue=clientId
I have tried adding proper timespans but it was of no use as there is no exception thrown for OperationTimedOut. Furthermore since I am new to ServiceFabric I maybe totally doing anything wrong.
PS: Sorry for maybe a lot of jumbled up and confused code and question AND SOME OF THE INFORMATION IS OBFUSCATED DUE TO CONFIDENTIALITY BUT NOTHING OBSTRUCTING THE UNDERSTANDING OF THIS IS HIDDEN (I Hope not an issue)
I hope this is not an issue maybe an error I am overlooking at my side
When you put the request in the queue, in AddToRequestToQueueAsync(), the name of the queue is set from _key.terminalId (and I don't see where you assign it), but when you read from it, in GetRequestJsonFromQueue(), the clientId
is used as the queue name.
I am using C# and AWSSDK v3 to upload files into an S3 bucket. The file is encrypted using ServerSideEncryptionCustomerMethod. I can upload the file, but if I check if the file exists using S3FileInfo().Exists, an error is thrown as a (400) Bad Request. However, if I comment out the lines that specify encryption in the upload routine, the S3FileInfo().Exists finds the file without throwing an error. What I am doing wrong? Or is there a different way to check if a file exists when it is encrypted?
Here is my upload routine:
public static string wfUpload(Stream pFileStream, string pBucketName, string pKeyName, string pCryptoKey) {
string retVal = "";
try {
using (var lS3Client = new AmazonS3Client()) {
Aes aesEncryption = Aes.Create();
aesEncryption.KeySize = 256;
aesEncryption.GenerateKey();
string lCryptoKey = Convert.ToBase64String(aesEncryption.Key);
PutObjectRequest request = new PutObjectRequest {
BucketName = pBucketName,
Key = pKeyName,
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = lCryptoKey,
};
request.InputStream = pFileStream;
PutObjectResponse response = lS3Client.PutObject(request);
retVal = lCryptoKey;
}
}
catch (AmazonS3Exception s3Exception) {
Console.WriteLine(s3Exception.Message,
s3Exception.InnerException);
throw (s3Exception);
}
catch (Exception e) {
throw (e);
}
return retVal;
}
And my routine to check if the file exists or not:
public static bool wfFileExists(String pBucketName, String pKeyName) {
bool retVal = false;
try {
using (var lS3Client = new AmazonS3Client()) {
if (new Amazon.S3.IO.S3FileInfo(lS3Client, pBucketName, pKeyName).Exists) {
retVal = true;
}
}
}
catch (AmazonS3Exception s3Exception) {
Console.WriteLine(s3Exception.Message,
s3Exception.InnerException);
throw (s3Exception);
}
catch (Exception e) {
throw (e);
}
return retVal;
}
Well, I think the class/method I was using is one of the high level APIs that doesn't support encryption. I changed my code to do a meta-data query to see if anything comes back. If it can't find the file it throws a "NotFound" ErrorCode in the s3Exception that I check for. Hopefully this helps someone else. If someone else suggests a better approach, I'd love to learn it too.
public static bool wfFileExists(String pBucketName, String pKeyName, String pCryptoKey) {
bool retVal = false;
try {
using (var lS3Client = new AmazonS3Client()) {
GetObjectMetadataRequest request = new GetObjectMetadataRequest {
BucketName = pBucketName,
Key = pKeyName,
ServerSideEncryptionCustomerMethod = ServerSideEncryptionCustomerMethod.AES256,
ServerSideEncryptionCustomerProvidedKey = pCryptoKey,
};
GetObjectMetadataResponse lMetaData = lS3Client.GetObjectMetadata(request);
// If an error is not thrown, we found the metadata.
retVal = true;
}
}
catch (AmazonS3Exception s3Exception) {
Console.WriteLine(s3Exception.Message,
s3Exception.InnerException);
if (s3Exception.ErrorCode != "NotFound") {
throw (s3Exception);
}
}
catch (Exception e) {
throw (e);
}
return retVal;
}
I'm having trouble figuring out how to determine if a number is duplicated.
Right now, the process is when the user clicks on a button to browse for an xml file, the xml file gets deserialized and stored into db and the data gets shown on a DataGrid on the view.
So, I added a confirmation dialog so when the user clicks on browse, the code checks to see if the lot_number being deserialized is a duplicate or not from inside a column from a table in database. I only want the user to be able to add lot numbers to db that are not duplicates.
Here's my code so far:
public void DeSerializationStream(string filePath)
{
XmlRootAttribute xRoot = new XmlRootAttribute();
xRoot.ElementName = "lot_information";
xRoot.IsNullable = false;
// Create an instance of lotinformation class.
var lot = new LotInformation();
// Create an instance of stream writer.
TextReader txtReader = new StreamReader(filePath);
// Create and instance of XmlSerializer class.
XmlSerializer xmlSerializer = new XmlSerializer(typeof(LotInformation), xRoot);
// DeSerialize from the StreamReader
lot = (LotInformation)xmlSerializer.Deserialize(txtReader);
// Close the stream reader
txtReader.Close();
}
public void ReadLot(LotInformation lot)
{
try
{
using (var db = new DDataContext())
{
var lotNumDb = db.LotInformation.FirstOrDefault(r => r.lot_number.Equals(r.lot_number));
if (lotNumDb != null || lotNumDb.lot_number.ToString().Equals(lot.lot_number))
{
confirmationWindow.Message = LanguageResources.Resource.Sample_Exists_Already;
dialogService.ShowDialog(LanguageResources.Resource.Error, confirmationWindow);
}
else {
Console.WriteLine("lot does not exist. yay");
}
DateTime ExpirationDate = lot.exp_date;
if (ExpirationDate != null)
{
if (DateTime.Compare(ExpirationDate, DateTime.Now) > 0)
{
try
{
LotInformation lotInfo = db.LotInformation.FirstOrDefault(r => r.lot_number.Equals(lotNumber));
}
catch (InvalidOperationException e)
{
//TODO: Add a Dialog Here
}
}
else
{
Console.WriteLine(ExpirationDate);
errorWindow.Message = LanguageResources.Resource.Lot_Expired;
dialogService.ShowDialog(LanguageResources.Resource.Error, errorWindow);
}
}
else
{
errorWindow.Message = LanguageResources.Resource.Lot_Not_In_Database;
dialogService.ShowDialog(LanguageResources.Resource.Error, errorWindow);
}
}
}
catch
{
errorWindow.Message = LanguageResources.Resource.Database_Error;
dialogService.ShowDialog(LanguageResources.Resource.Error, errorWindow);
logger.writeErrLog(LanguageResources.Resource.Database_Error);
}
}
I think I'm just having problems with when to grab the lot_number in this process.
This part below gives me problems. It keeps showing the Sample Exists already message for unique lot numbers that I'm uploading and I'm not sure why. I think it's a problem with my LINQ query but I'm not sure how to fix it. Any ideas?
var lotNumDb = db.LotInformation.FirstOrDefault(r => r.lot_number.Equals(r.lot_number));
if (lotNumDb != null || lotNumDb.lot_number.ToString().Equals(lot.lot_number))
{
confirmationWindow.Message = LanguageResources.Resource.Sample_Exists_Already;
dialogService.ShowDialog(LanguageResources.Resource.Error, confirmationWindow);
}
else {
Console.WriteLine("lot does not exist. yay");
}
you can't use like this:
db.LotInformation.FirstOrDefault(r => r.lot_number.Equals(r.lot_number))
may be :
db.LotInformation.FirstOrDefault(r => r.lot_number.Equals(lot.lot_number))
or
db.LotInformation.FirstOrDefault(r => r.lot_number.Equals(a string))
I am trying to timeout and throw an exception after waiting a specified amount of time, and am wondering if the way I'm currently doing it is best.
class Timeout
{
XmlDocument doc;
System.Object input;
public Timeout(XmlDocument doc, System.Object input)
{
this.doc = doc;
this.input = input;
}
public void run()
{
if (input is Stream)
{
doc.Load((Stream)input);
}
else if (input is XmlReader)
{
doc.Load((XmlReader)input);
}
else if (input is TextReader)
{
doc.Load((TextReader)input);
}
else
{
doc.Load((string)input);
}
System.Threading.Thread.CurrentThread.Abort();
}
}
private void LoadXmlDoc(XmlDocument doc, System.Object input)
{
Timeout timeout = new Timeout(doc, input);
System.Threading.Thread timeoutThread = new System.Threading.Thread(new ThreadStart(timeout.run));
timeoutThread.Start();
System.Threading.Thread.Sleep(this.timeout * 1000);
if (timeoutThread.IsAlive)
{
throw new DataSourceException("timeout reached", timeout.GetHashCode());
}
}
This current approach does work, so I'm just wondering if there's a simpler/better way to go about accomplishing the same thing.
In addition to my comment (here's the link from it), here is some more information regarding threading. Basically it comes down to learning the different designs/libraries, what their pros and cons are, which one suits your needs best.
From my understanding, and hopefully someone with more knowledge on the subject will pitch in on this, there are basically 2 different categories that you can put the threading designs in, synchronous and asynchronous. You have used the asynchronous design, employing the thread pool. If you want to stick with this design, you can try using the Task or, for synchronous operations Parallel.
On a side note: I'm not sure as to the wisdom behind using an exception to handle simple logic. In other words, the exception could be simply replace with returning a boolean.
Try doing something like this:
try
{
var cts = new System.Threading.CancellationTokenSource();
cts.CancelAfter(TimeSpan.FromSeconds(0.01));
var tw = Task.Run<System.Xml.XmlDocument>(() =>
{
var doc = new System.Xml.XmlDocument();
doc.Load("https://maps.googleapis.com/maps/api/geocode/xml?address=1+Exchange+Plaza+,+Floor+26+,+NY&sensor=false");
return doc;
}, cts.Token);
var xml = await tw;
}
catch (TaskCanceledException e)
{
}
What I ended up doing:
class Timeout
{
XmlDocument doc;
System.Object input;
public Timeout(XmlDocument doc, System.Object input)
{
this.doc = doc;
this.input = input;
}
public void run()
{
if (input is Stream)
{
doc.Load((Stream)input);
}
else if (input is XmlReader)
{
doc.Load((XmlReader)input);
}
else if (input is TextReader)
{
doc.Load((TextReader)input);
}
else
{
doc.Load((string)input);
}
}
}
private void LoadXmlDoc(XmlDocument doc, System.Object input)
{
Timeout timeout = new Timeout(doc, input);
System.Threading.Thread timeoutThread = new System.Threading.Thread(new ThreadStart(timeout.run));
timeoutThread.Name = "XmlWorker" + threadNumber++;
timeoutThread.Start();
if (!timeoutThread.Join(this.timeout)) //Join returning false implies the timeout was reached
{
if (timeoutThread.IsAlive)
timeoutThread.Abort();
throw new DataConnectionException("timeout reached: " + this.timeout.Milliseconds + "ms", new TimeoutException(this.timeout.Milliseconds));
}
}