I have two reliable queues and they are being accessed by two guest executables and each of them access their own. Sometimes the function I use to access them doesn't update the reliable queue object in the function and the wrong request is sent to the wrong guest executable.
What happens is that the clientId is passed by the guest executable to this function in the Get request. Let us say that there are two clientId(s) called T1 and T2.
What happens is that the guest executable (client) T2 at times gets the request that was meant for T1. Even though I tried line by line debugging the parameters passed to this function are correct.
Here is my API's POST that is passed a json to be added to the queue for the clients to receive from the GET
[HttpPost("MarketInfo")]
public JObject GetMarketInfo([FromBody] JObject jObject)
{
List<JToken> clients = jObject.GetValue("clients").ToList();
string json;
JObject response = new JObject();
JArray jsonArray = new JArray();
try
{
foreach (JToken client in clients)
{
var id = Guid.NewGuid();
json = "{'name':'MarketInfo','id':'" + id.ToString() + "','mtClientId':'" + terminal["name"].ToString() + "','parameters':{'symbol':'" + terminal["symbol"].ToString() + "','property':24}}";
bool result = _requestsCollectionHandler.CreateRequestForClient(JObject.Parse(json));
JObject clientResponse = new JObject();
if (result==true)
{
clientResponse["name"] = client["name"].ToString();
clientResponse["guid"] = id.ToString();
jsonArray.Add(clientResponse);
}
else
{
clientResponse["name"] = terminal.Children()["name"].ToString();
clientResponse["guid"] = "ERROR";
jsonArray.Add(terminalResponse);
}
}
response["clients"] = jsonArray;
return response;
}
catch (Exception e)
{
Debug.Write(e.Message);
return null;
}
}
This is the json that we pass to this API
{"clients":[{"name":"T1","symbol":"SomeInfo"},{"name":"T2","symbol":"SomeInfo"}]}
The problem is always with the clients object that is passed first.
Before I explain further let me also share the code for the client's HttpGet
[HttpGet("{clientId}")]
public string Get([FromRoute] string clientId)
{
try
{
string request = _requestsCollectionHandler.GetRequestJsonFromQueue(clientId);
return request;
}
catch(Exception e)
{
return e.Message;
}
}
This is the function that creates an object that is to be added by another function in the reliable queue
public bool CreateRequestForClient(JObject jObject)
{
try
{
this._jObject = new JObject(jObject);
CreateKey();
AddToRequestToQueueAsync();
return true;
}
catch (Exception e)
{
Debug.Write(e.Message);
_exceptionMessage = e.Message;
return false;
}
}
private void CreateKey()
{
dynamic data = JObject.Parse(_jObject.ToString(Newtonsoft.Json.Formatting.None));
string name = data.name;
string id = data.id;
string clientId = data.clientId;
_key.id = id;
_key.name = name;
_key.clientId = clientId;
//key.timestamp = GetTimestamp();
_key.timestamp = GetTimestamp();
_key.requestJson = _jObject.ToString(Newtonsoft.Json.Formatting.None);
}
_key is a private variable in class a custom class
This is the function in my class of request handler that adds the requests to the queue
private void AddToRequestToQueueAsync()
{
var transaction = this._stateManager.CreateTransaction();
CancellationToken cancellationToken
= new CancellationToken(false);
try
{
string queue = _key.clientId;
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
_stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
transaction = this._stateManager.CreateTransaction();
if (reliableQueue!=null)
{
long count = reliableQueue.Count;
reliableQueue.EnqueueAsync(transaction, _key);
count = reliableQueue.Count;
transaction.CommitAsync().Wait();
}
else
{
transaction.Abort();
}
}
catch
{
transaction.Abort();
throw;
}
}
This is function that is used by the client
public string GetRequestJsonFromQueue(string clientId)
{
string queue = clientId;
try
{
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
if(reliableQueue != null)
{
ConditionalValue<TerminalResponseKey> key =
reliableQueue.TryDequeueAsync(transaction).Result;
if(key.HasValue)
{
string request = key.Value.requestJson;
transaction.CommitAsync().Wait();
return request;
}
}
else
{
transaction.Abort();
}
return "NO QUEUE";
}
catch (Exception e)
{
Debug.WriteLine(e);
transaction.Abort();
return e.InnerException.Message;
}
}
As far as I have found out I think my problem is in this function above. Because I don't know how the client T2 or client T1 gets another client's queue because the parameters determining the queue are their IDs and are totally unique.
These Ids are also passed correctly to this:
IReliableConcurrentQueue<TerminalResponseKey> reliableQueue =
this._stateManager.GetOrAddAsync<IReliableConcurrentQueue<TerminalResponseKey>>(queue).Result;
As you can see that we have queue=clientId
I have tried adding proper timespans but it was of no use as there is no exception thrown for OperationTimedOut. Furthermore since I am new to ServiceFabric I maybe totally doing anything wrong.
PS: Sorry for maybe a lot of jumbled up and confused code and question AND SOME OF THE INFORMATION IS OBFUSCATED DUE TO CONFIDENTIALITY BUT NOTHING OBSTRUCTING THE UNDERSTANDING OF THIS IS HIDDEN (I Hope not an issue)
I hope this is not an issue maybe an error I am overlooking at my side
When you put the request in the queue, in AddToRequestToQueueAsync(), the name of the queue is set from _key.terminalId (and I don't see where you assign it), but when you read from it, in GetRequestJsonFromQueue(), the clientId
is used as the queue name.
Related
So, i'm calling the method to update the primary "sendAs" object of a google account, without results. The documentation from google at users.settings.sendAs/update indicates all i need and did:
i've set the domain wide account and scopes
i'm generating a token and accessing it no problem
i'm calling the "list" method first (with that token) as shown in users.settings.sendAs/list documentation, and finding the one that is the primary (the "isPrimary" attribute is true)
After that, changing the "signature" value to "Its a Test Signature", and sending the PUT request with it doesn't do anything.
The JSON sent to the update API (via PUT method) is the exact one i collected from the list (the primary one), but with the signature changed.
There is no error at all, and i receive an "sendAs" object back as a response (as the documentation says i should in case of sucess), but the signature is unchanged.
What can i be?
EDIT (adding the code section for the call, again - no errors)
public bool Update()
{
string json = null;
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.ContentType] = "application/json";
wc.Headers["Authorization"] = "Bearer " + GenerateServerToServerToken(this.OwnerMail, this.scope);
json = wc.DownloadString("https://gmail.googleapis.com/gmail/v1/users/" + this.OwnerMail + "/settings/sendAs");
JSon.Query response = JSon.Parse(ref json);
json = null;
response = response["sendAs"];
List<JSon.Query> mailAs = null;
if (response.TryParseList(out mailAs))
{
JSon.Query main = null;
bool prim = false;
foreach (JSon.Query q in mailAs)
{
if (q["isPrimary"].TryParseBoolean(out prim) && prim)
{
if (q["sendAsEmail"].TryParseString(out json)) { main = q; }
break;
} else { json = null; }
}
if (main != null)
{
JSon.ObjectValue mainO = (JSon.ObjectValue)main.Value;
if (mainO.ContainsKey("signature"))
{
((JSon.StringValue)mainO["signature"]).Data = this.HtmlSignature.Replace("<", ("\\" + "u003c")).Replace(">", ("\\" + "u003e"));
mainO["verificationStatus"] = new MdIO.JSon.StringValue("accepted");
json = wc.UploadString("https://gmail.googleapis.com/gmail/v1/users/" + this.OwnerMail + "/settings/sendAs/" + json, "PUT", main.Value.ToJSON());
response = JSon.Parse(ref json);
if (response["sendAsEmail"].TryParseString(out json) && !string.IsNullOrEmpty(json)) { return true; }
}
}
}
}
return false;
}
I have a lambda built with ApiGateway / Serverless.
It will execute a MySQL query. The MySQL Server is up and running and performing other queries with no issues, it seems that this is the only Lambda that returns an empty object.
It accepts a post request with a ID number in the body and then queries the DB and returns the object/data. Like before the server is running correctly. Just this returns and empty object.
I started to think that this maybe completing before the results were returned, I tried running this both sync and aync with the same results.
I am newish to c# so forgive if the code could be more efficient.
No clue as to why this returns a empty object everytime.
namespace Aws.Lambda
{
public class SearchHandler:BaseHandler
{
public async Task<APIGatewayProxyResponse> getId(LambdaRequest request)
{
var error = new Errorhandle();
var connect = new Connection(); //THIS IS THE DATABASE CONNECTION CLASS (WORKS)
var result = new List<ReturnTypes>();
var number = JsonConvert.DeserializeObject<bodyType>(request.body);
bool boolNumber = String.IsNullOrWhiteSpace(number.factura);
try
{
if(boolNumber)
{
return error.errorApiResponse("empty");
}
else
{
await Task.Yield(); // TRYING TO ASYNC / AWAIT THIS,
string x = $"SELECT * FROM Table1 WHERE ID_NUM ='{number}'";
Console.WriteLine(x); //LOGGING THE QUERY FOR ERRORS (NONE FOUND COPY AND PASTED QUERY AND IT WORKED IN WORKBENCH).
var results = connect.getData(x); //QUERY EXECUTE PASS TO FUNCTION MADE IN CLASS
if(!results.Read())
{
return error.errorApiResponse("reader is empty");
}
while (results.Read())
{
resultList.Add(insertData(results));
}
return new APIGatewayProxyResponse
{
StatusCode = 200,
Body = JsonConvert.SerializeObject(resultList),
Headers = headerList = new Dictionary<string, string>
{
{ "Content-Type", "application/json" },
{ "Access-Control-Allow-Origin", "*" },
{"Access-Control-Allow-Methods","OPTIONS,GET"}
};
};
}
} catch (Exception e)
{
return error.errorApiResponse("exception error");
throw e;
}
}
private static ReturnTypes insertData(System.Data.IDataReader results)
{
return new ReturnTypes { Num = results["ID"], Body = results["UUID"];
}
}
}
Im creating a on-prem application in .Net that consumes a kinesis stream.
How do I write a method that can renew the cridentials for my KinesisClient (They are only valid for 1 hour) without re-create the client itself. Is that possible? Or am I approching this the wrong way?
This is the factory class that im using to produce the client.
public static AmazonKinesisClient CreateKinesisClient(BasicAWSCredentials credentials, string roleArn, string roleSessionName, RegionEndpoint region, LogWriter logWriter)
{
try
{
var stsClient = new AmazonSecurityTokenServiceClient(credentials);
var ownerRoleReq = new AssumeRoleRequest
{
RoleArn = roleArn,
RoleSessionName = roleSessionName
};
var ownerRoleResp = stsClient.AssumeRoleAsync(ownerRoleReq).Result;
logWriter.LogInfo<AmazonKinesisClient>("Role taken:" + ownerRoleResp.AssumedRoleUser);
return new AmazonKinesisClient(ownerRoleResp.Credentials, region);
}
catch (Exception e)
{
logWriter.LogError<AmazonClientFactory>(e.Message,e);
throw;
}
}
I ended up with saving the latest record id in a concurrent dictionary and while re-creating the client injecting the dictionary containing the latest id for each shard in the KinesisStream.
private string CreateShardIteratorRequestWithSequenceNumber(Shard shard, string sequenceNumber)
{
var iteratorRequest = new GetShardIteratorRequest()
{
ShardId = shard.ShardId,
StreamName = _streamName,
ShardIteratorType = ShardIteratorType.AFTER_SEQUENCE_NUMBER,
StartingSequenceNumber = sequenceNumber
};
return GetShardIterator(iteratorRequest);
}
I have written an app that goes through our own properties and scraps the data. To make sure I don't run through the same URLs, I am using a MySQL database to store the URL, flag it once its processed. All this was being done in a single thread and it's fine if I had only few thousand entries. But I have few hundred thousand entries that I need to parse so I need to make changes in the code (I am newbie in multithreading in general). I found an example and was trying to copy the style but doesn't seem to work. Anyone know what the issue is with the following code?
EDIT: Sorry didn't mean to make people guess the issue but was stupid of me to include the exception. Here is the exception
"System.InValidCastException: 'Specified cast is not valid.'"
When I start the process it collects the URLs from the database and then never hits DoWork method
//This will get the entries from the database
List<Mappings> items = bot.GetUrlsToProcess(100);
if (items != null)
{
var tokenSource = new CancellationTokenSource();
var token = tokenSource.Token;
Worker.Done = new Worker.DoneDelegate(WorkerDone);
foreach (var item in items)
{
urls.Add(item.Url);
WaitingTasks.Enqueue(new Task(id => new Worker().DoWork((int)id, item.Url, token), item.Url, token));
}
LaunchTasks();
}
static async void LaunchTasks()
{
// keep checking until we're done
while ((WaitingTasks.Count > 0) || (RunningTasks.Count > 0))
{
// launch tasks when there's room
while ((WaitingTasks.Count > 0) && (RunningTasks.Count < MaxRunningTasks))
{
Task task = WaitingTasks.Dequeue();
lock (RunningTasks) RunningTasks.Add((int)task.AsyncState, task);
task.Start();
}
UpdateConsole();
await Task.Delay(300); // wait before checking again
}
UpdateConsole(); // all done
}
static void UpdateConsole()
{
Console.Write(string.Format("\rwaiting: {0,3:##0} running: {1,3:##0} ", WaitingTasks.Count, RunningTasks.Count));
}
static void WorkerDone(int id)
{
lock (RunningTasks) RunningTasks.Remove(id);
}
public class Worker
{
public delegate void DoneDelegate(int taskId);
public static DoneDelegate Done { private get; set; }
public async void DoWork(object id, string url, CancellationToken token)
{
if (token.IsCancellationRequested) return;
Content obj;
try
{
int tries = 0;
bool IsUrlProcessed = true;
DateTime dtStart = DateTime.Now;
string articleDate = string.Empty;
try
{
ScrapeWeb bot = new ScrapeWeb();
SearchApi searchApi = new SearchApi();
SearchHits searchHits = searchApi.Url(url, 5, 0);
if (searchHits.Hits.Count() == 0)
{
obj = await bot.ReturnArticleObject(url);
if (obj.Code != HttpStatusCode.OK)
{
Console.WriteLine(string.Format("\r Status is {0}", obj.Code));
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.HttpCode = obj.Code;
}
else
{
string title = obj.Title;
string content = obj.Contents;
string description = obj.Description;
Articles article = new Articles();
article.Site = url.GetSite();
article.Content = content;
article.Title = title;
article.Url = url.ToLower();
article.Description = description;
string strThumbNail = HtmlHelper.GetImageUrl(url, obj.RawResponse);
article.Author = HtmlHelper.GetAuthor(url, obj.RawResponse);
if (!string.IsNullOrEmpty(strThumbNail))
{
//This condition needs to be added to remove ?n=<number> from EP thumbnails
if (strThumbNail.Contains("?"))
{
article.ImageUrl = strThumbNail.Substring(0, strThumbNail.IndexOf("?")).Replace("http:", "https:");
}
else
article.ImageUrl = strThumbNail.Replace("http:", "https:");
}
else
{
article.ImageUrl = string.IsNullOrEmpty(strThumbNail) ? article.Url.GetDefaultImageUrls() : strThumbNail.Replace("http:", "https:");
}
articleDate = HtmlHelper.GetPublishDate(url, obj.RawResponse);
if (string.IsNullOrEmpty(articleDate))
article.Pubdate = DateTime.Now;
else
article.Pubdate = DateTime.Parse(articleDate);
var client = new Index(searchApi);
var result = client.Upsert(article);
itemfound.HttpCode = obj.Code;
if (result)
{
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate);
UpdateItem(itemfound);
}
else
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
UpdateItem(itemfound, tries, IsUrlProcessed);
}
}
}
else
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = true;
itemfound.HttpCode = HttpStatusCode.OK;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
}
}
catch (Exception e)
{
tries = itemfound.UrlMaxTries + 1;
IsUrlProcessed = false;
itemfound.DateCreated = DateTime.Parse(articleDate);
itemfound.DateModified = DateTime.Parse(articleDate) == null ? DateTime.Now : DateTime.Parse(articleDate);
}
finally
{
DateTime dtEnd = DateTime.Now;
Console.WriteLine(string.Format("\r Total time taken to process items is {0}", (dtEnd - dtStart).TotalSeconds));
}
}
catch (Exception e)
{
Console.WriteLine(e);
}
Done((int)id);
}
}
All this code is based from Best multi-thread approach for multiple web requests this link. Can someone tell me how to get this approach running?
I think the problem is in the way you're creating your tasks:
new Task(id => new Worker().DoWork((int)id, item.Url, token), item.Url, token)
This Task constructor overload expected Action<object> delegate. That means id will be typed as object and you need to cast it back to something useful first.
Parameters
action
Type: System.Action<Object>
The delegate that represents the code to execute in the task.
state
Type: System.Object
An object representing data to be used by the action.
cancellationToken
Type: System.Threading.CancellationToken
-The CancellationToken that that the new task will observe.
You decided to cast it to int by calling (int)id, but you're passing item.Url as the object itself. I can't tell you 100% what the type of Url is but I don't expect Url-named property to be of type int.
Based on what #MarcinJuraszek said I just went back to my code and added an int as I couldn't find another way to resolve it. Here is the change I made
int i=0
foreach (var item in items)
{
urls.Add(item.Url);
WaitingTasks.Enqueue(new Task(id => new Worker().DoWork((string)id, item.Url, token), item.Url, token));
i++;
}
Scenario
One windows service polls a url every two minutes to retrieve certain data.
If any data has been added since the previous call, the data is retrieved and stored otherwise the loop carries on.
Issue
Sometimes a request takes more than two minutes to return a response.
When this happens, the next request is still made and finds new data, since the previous request hasn't return a response yet
This results in duplicate entries when the data is stored.
What I've tried
I tried to handle that by using a boolean like so:
Boolean InProgress = true;
foreach (var item in Lists)
{
\\Make a request and return new data (if any)
InProgress = false;
if (InProgress = false)
{
\\Store new data
}
}
This doesn't solve the issue. I believe I'm using the boolean in wrong place, but I'm not sure where it should.
This is the loop that makes the request and store the data
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
try
{
Data getCredentials = new Data();
DataTable credentials = getCredentials.loadCredentials();
Boolean InProgress = true;
for (int i = 0; i < credentials.Rows.Count; i++)
{
if (credentials != null)
{
var PBranchID = (int)credentials.Rows[i]["PortalBranchID"];
var negRef = (int)credentials.Rows[i]["NegotiatorRef"];
var Username = credentials.Rows[i]["Username"].ToString();
var Password = credentials.Rows[i]["Password"].ToString();
var Domain = credentials.Rows[i]["Domain"].ToString();
var FooCompanyBaseUrl = "https://" + Domain + ".FooCompany.com/";
Data getCalls = new Data();
DataTable calls = getCalls.loadCalls(PBranchID);
//If it's not the first call
if (calls != null && calls.Rows.Count > 0)
{
//Makes a call
DateTime CreatedSince = DateTime.SpecifyKind((DateTime)calls.Rows[0]["LastSuccessOn"], DateTimeKind.Local);
string IssueListUrl = FooCompany.WebApi.V2.URLs.Issues(BaseUrl, null, CreatedSince.ToUniversalTime(), null);
FooCompany.WebApi.V2.DTO.PrevNextPagedList resultIssueList;
resultIssueList = FooCompany.WebApi.Client.Helper.Utils.Getter<Foocompany.WebApi.V2.DTO.PrevNextPagedList>(IssueListUrl, Username, Password);
InProgress = false;
if (InProgress == false)
{
if (resultIssueList.Items.Count > 0)
{
//If call returns new issues, save call
Data saveCalls = new Data();
saveCalls.saveCalls(PBranchID);
foreach (var item in resultIssueList.Items)
{
var Issue = FooCompany.WebApi.Client.Helper.Utils.Getter<FooCompany.WebApi.V2.DTO.Issue>(item, Username, Password);
string TenantSurname = Issue.Surname;
string TenantEmail = Issue.EmailAddress;
Data tenants = new Data();
int tenantPropRef = Convert.ToInt32(tenants.loadTenantPropRef(PBranchID, TenantSurname, TenantEmail));
Data Properties = new Data();
DataTable propAddress = Properties.loadPropAddress(PBranchID, tenantPropRef);
var Address1 = propAddress.Rows[0]["Address1"];
var Address2 = propAddress.Rows[0]["Address2"];
var AddressFolder = Address1 + "," + Address2;
if (!Directory.Exists("path"))
{
Directory.CreateDirectory("path");
}
string ReportPDFDestination = "path";
if (File.Exists(ReportPDFDestination))
{
File.Delete(ReportPDFDestination);
}
FooCompany.WebApi.Client.Helper.Utils.DownloadFileAuthenticated(FooCompany.WebApi.V2.URLs.IssueReport(BaseUrl, Issue.Id), Username, Password, ReportPDFDestination);
//Store data
}
IssueListUrl = resultIssueList.NextURL;
}
}
}
else
{
continue;
}
}
}
catch (Exception ex)
{
//write to log
}
}
Question
I'm sure there is a better way than a boolean.
Could anyone advice a different method to handle the issue properly?
Thanks.
Solution
I ended up using a combination of both Thomas and Mason suggestions. I wrapped a lock statement around the main function of my windows service and used a boolean inside the function section that makes the call to the remote server.
Tested many times and it's error free.
You seems to have a problem of synchronisation, just surround the code that iterate though the List with a lock, and you will be fine.
public class MyClass{
private readonly object internalLock= new object();
private bool AlreadyRunning { get; set; }
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(AlreadyRunning){
return;
}
try{
lock(internalLock){
Thread.MemoryBarrier();
if(AlreadyRunning){
return;
}
AlreadyRunning = true;
...Do all the things...
}
}
catch(Exception e){
..Exception handling
}
finally
{
AlreadyRunning = false;
}
}
bool InProgress=false;
void serviceTimer_Elapsed(object sender, ElapsedEventArgs e)
{
if(!InProgress)
{
InProgress=true;
//retrieve data
InProgress=false;
}
}
Your InProgress variable needs to be declared outside the event handler. When you enter the method, check to see if it's already running. If it is, then we do nothing. If it's not running, then we say it's running, retrieve our data, then reset our flag to say we've finished running.
You'll probably need to add appropriate locks for thread safety, similar to Thomas's answer.