I'm using TLSharp. My goal is to send files to the user. I created ASP.NET Core Web API service and make HTTP request when I need to send file.
It works well with one file but every time when I get 2 or more requests in a short period of time I get an error:
System.InvalidOperationException: invalid checksum! skip.
Controller:
[Route("upload/{driveId}")]
public async Task<ActionResult> Upload(string driveId)
{
var ms = new MemoryStream();
var file = service.Files.Get(driveId);
string filename = file.Execute().Name;
await file.DownloadAsync(ms);
ms.Position = 0;
new FileExtensionContentTypeProvider().TryGetContentType(filename, out var mime);
var stream = new StreamReader(ms, true);
await _client.SendFileToBot(filename, mime, stream, driveId);
return Ok();
}
SendFileToBot method:
public async Task SendFileToBot(string filename, string mime, StreamReader stream)
{
var found = await client.SearchUserAsync("username", 1);
//find user
var userToSend = found.Users
.Where(x => x.GetType() == typeof(TLUser))
.Cast<TLUser>()
.FirstOrDefault(x => x.Id == 1234567);
var fileResult = await client.UploadFile(filename, stream);
var attr = new TLVector<TLAbsDocumentAttribute>()
{
new TLDocumentAttributeFilename { FileName = filename }
};
var bot = new TLInputPeerUser() { UserId = userToSend.Id, AccessHash = userToSend.AccessHash.Value };
await client.SendUploadedDocument(bot, fileResult, "caption", mime, attr);
}
When the requests are sent together (or in short period of time), they're sent in a single packet to Telegram server and this error occurs. I need help with this error. I've tried to use Task.Delay but it doesn't help.
How can I handle requests to avoid this error?
According this issue, you are not first person who received this error.
Seems like there are something request/response validation issues when using multithreading in TLSharp library.
There is one stable workaround for such type of problems.
Make all upload requests synchronous
Actually, they will be asynchronous, but with one-task-at-one-time access
This dirty but workable solution can be achieved by creating task queue:
public class TaskQueue
{
private readonly SemaphoreSlim _semaphoreSlim;
public TaskQueue()
{
_semaphoreSlim = new SemaphoreSlim(1, 1); // Max threads limited to 1.
}
public async Task<T> Enqueue<T>(Func<Task<T>> taskGenerator)
{
await _semaphoreSlim.WaitAsync();
try
{
return await taskGenerator();
}
finally
{
_semaphoreSlim.Release();
}
}
public async Task Enqueue(Func<Task> taskGenerator)
{
await _semaphoreSlim.WaitAsync();
try
{
await taskGenerator();
}
finally
{
_semaphoreSlim.Release();
}
}
}
Now you must register queue as singleton in Startup.cs file to be sure that your asp.net core application using one task queue instance to perform uploading on telegram servers:
public void ConfigureServices(IServiceCollection services)
{
//...
services.AddSingleton<TaskQueue>();
//...
}
Next, get your task queue instance in your api controller constructor like:
private readonly TaskQueue taskQueue;
public MyController(TaskQueue taskQueue)
{
this.taskQueue = taskQueue
}
Then, just use it in all of your api methods:
// Code from your API method...
await taskQueue.Enqueue(() => client.SendUploadedDocument(bot, fileResult, "caption", mime, attr));
This will make all requests to telegram servers through TLSharp library synchronous and prevent multithreading issues like in a question.
To be honest, it's just a workaround, not solution of this problem. I'm sure that this issue on github about checksum error must be investigated more detailed and fixed if it possible.
Related
Application : Asp.Net Core Web API
I have requirement, On receiving the notification I should do 2 independent tasks. notification gets triggered as apart of web hook.
call another service and then saves the response in log.
Since this is independent task I do not want to wait the caller until we receive response. So I thought to wrap this function in Task.Run(), so that it will be run on another thread available on thread pool.
I donot care wheather it fails or success, I just need to log the response.
Task.Run(() => ProcessThroughRestClient(this.httpContext.Request, this.cspgConfiguration.Value.ApacNotificationService, this.restClient, this.logger));
Saves the request object in DB for tracing.
Since I must save the notification, I made it as awaitable task.
await this.invoiceDbInstance.UpdateItemAsync(payment);
Below is the full code.
Main Method
public async Task<IActionResult> NotifyAsync()
{
this.logger.LogTrace("{CorrelationId} - notify async method has been called.", this.CorrelationId);
Task.Run(() => ProcessThroughRestClient(this.httpContext.Request, this.Configuration.Value.NotificationService, this.restClient, this.logger));
var request = this.GetDataFeedRequest(this.httpContext.Request);
if (request != null)
{
this.logger.LogInformation("{CorrelationId} - data feed invoked with the order reference {OrderReference}", request.OrderReference, this.CorrelationId);
Func<Invoice, bool> condition = x => x.SourceTransactionId == request.OrderReference;
var payment = this.invoiceDbInstance.GetItem(condition);
if (payment != null)
{
payment.TransactionStatus = request.IsSuccessStatusCode() ? TransactionStatus.Success : TransactionStatus.Failure;
payment.TransactionId = request.PaymentReference;
payment.UpdatedAt = DateTime.UtcNow.ToLongDateString();
await this.invoiceDbInstance.UpdateItemAsync(payment);
this.logger.LogInformation("{CorrelationId} - data feed updated order reference {OrderReference} updated in Invoice with {PaymentReference}", request.OrderReference, this.CorrelationId, request.PaymentReference);
}
else
{
this.logger.LogInformation("{CorrelationId} - Payment not found.", this.CorrelationId);
}
}
this.logger.LogTrace("{CorrelationId}- notify async method ended.", this.CorrelationId);
return new OkResult();
}
Http Call function which will be invoked through Task.Run()
private static HttpResponseMessage ProcessThroughRestClient(HttpRequest request, string url, IRestClient client, ILogger<PayDollarNotificationService> log)
{
try
{
log.LogTrace("Paydollar notify async ProcessThroughRestClient method has been called.");
var parameters = request.Form?.ToDictionary(x => x.Key, x => x.Value.ToString());
if (parameters.Any())
{
var response = client.PostAsync(url, RestClientHelper.BuildFormUrlEncodedContent(parameters)).Result;
log.LogInformation("sent request to {url}.", url);
log.LogInformation($"{url} response {response.ReadContent()?.Result}");
return response;
}
else
{
log.LogInformation("No form parameters found.");
return null;
}
}
catch (Exception ex)
{
log.LogError($"An error occured: {ex.Message} {ex}");
}
return null;
}
My question is, Is there any advantage using Task.Run() as above instead awitable task? or is it is going to be blocking thread?
I'm attempting to stream a large JSON file built on the fly to a client (could be 500 MB+). I'm trying to disable response buffering for a variety of reasons, though mostly for memory efficiency.
I've tried writing directly to the HttpContext.Response.BodyWriter but the response seems to be buffered in memory before writing to the output. The return type of this method is Task.
HttpContext.Response.ContentType = "application/json";
HttpContext.Response.ContentLength = null;
await HttpContext.Response.StartAsync(cancellationToken);
var bodyStream = HttpContext.Response.BodyWriter.AsStream(true);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("["), cancellationToken);
await foreach (var item in cursor.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
await bodyStream.WriteAsync(JsonSerializer.SerializeToUtf8Bytes(item, DefaultSettings.JsonSerializerOptions), cancellationToken);
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes(","), cancellationToken);
await bodyStream.FlushAsync(cancellationToken);
await Task.Delay(100,cancellationToken);
}
await bodyStream.WriteAsync(Encoding.UTF8.GetBytes("]"), cancellationToken);
bodyStream.Close();
await HttpContext.Response.CompleteAsync().ConfigureAwait(false);
Note: I realize this code is very hacky, trying to make it work, then clean it up
I'm using the Task.Delay to verify the response is not being buffered when testing locally as I do not have full production data. I have also tried IAsyncEnumerable and yield return, but that fails because the response is so large that Kestrel thinks the enumerable is infinite.
I've tried
Setting KestrelServerLimits.MaxResponseBufferSize to a small number, even 0;
Writing with HttpContext.Response.WriteAsync
Writing with HttpContext.Response.BodyWriter.AsStream()
Writing with a pipe writer patter and HttpContext.Response.BodyWriter
Removing all middleware
Removing calls to IApplicationBuilder.UseResponseCompression
Update
Tried disabling response buffering before setting the ContentType (so before any writes to the response) with no effect
var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();
Updated Sample Code
This reproduces the issue quite simply. The client doesn't receive any data until response.CompleteAsync() is called.
[HttpGet]
[Route("stream")]
public async Task<EmptyResult> FileStream(CancellationToken cancellationToken)
{
var response = DisableResponseBuffering(HttpContext);
HttpContext.Response.Headers.Add("Content-Type", "application/gzip");
HttpContext.Response.Headers.Add("Content-Disposition", $"attachment; filename=\"player-data.csv.gz\"");
await response.StartAsync().ConfigureAwait(false);
var memory = response.Writer.GetMemory(1024*1024*10);
response.Writer.Advance(1024*1024*10);
await response.Writer.FlushAsync(cancellationToken).ConfigureAwait(false);
await Task.Delay(5000).ConfigureAwait(false);
var str2 = Encoding.UTF8.GetBytes("Bar!\r\n");
memory = response.Writer.GetMemory(str2.Length);
str2.CopyTo(memory);
response.Writer.Advance(str2.Length);
await response.CompleteAsync().ConfigureAwait(false);
return new EmptyResult();
}
private IHttpResponseBodyFeature DisableResponseBuffering(HttpContext context)
{
var responseBufferingFeature = context.Features.Get<IHttpResponseBodyFeature>();
responseBufferingFeature?.DisableBuffering();
return responseBufferingFeature;
}
I was able to get this working when using http.sys (with ASP.NET Core 6):
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Features;
public class Program
{
public static void Main(string[] args)
{
var builder = WebApplication.CreateBuilder(args);
builder.WebHost.UseHttpSys();
var app = builder.Build();
app.MapGet("/", async (context) =>
{
context.Response.StatusCode = 201;
await context.Response.StartAsync();
await context.Response.WriteAsync("x"); // client gets status code after this line
await context.Response.WriteAsync("Hello World!");
});
app.Run();
}
}
Try to disable buffering on response futures:
HttpContext.Features.Get<IHttpResponseBodyFeature>().DisableBuffering()
//As mentioned in documentation, to take effect, call it before any writes
And use BodyWriter in Utf8JsonWriter for more efficiency:
var pipe = context.HttpContext.Response.BodyWriter;
await pipe.WriteAsync(startArray);
using (var writer = new Utf8JsonWriter(pipe,
new JsonWriterOptions
{
Indented = option.WriteIndented,
Encoder = option.Encoder,
SkipValidation = true
}))
{
var dotSet = false;
foreach (var item in enumerable)
{
if (dotSet)
await pipe.WriteAsync(dot);
JsonSerializer.Serialize(writer, item, itemType, option);
await pipe.FlushAsync();
writer.Reset();
dotSet = true;
}
}
await pipe.WriteAsync(endArray);
In my case it give results: total memory allocation become greater over 80% compared to newcoreapp2.2 after first requests, but no more memory leaks.
For those who is still interested this code sends data right away when using curl:
public async Task Invoke(HttpContext context)
{
var g = context.Features.Get<IHttpResponseBodyFeature>();
g.DisableBuffering(); // doesn't seem to make a difference
context.Response.StatusCode = 200;
context.Response.ContentType = "text/plain; charset=utf-8";
//context.Response.ContentLength = null;
await g.StartAsync();
for (int i = 0; i < 10; ++i)
{
var line = $"this is line {i}\r\n";
var bytes = utf8.GetBytes(line);
// it seems context.Response.Body.WriteAsync() and
// context.Response.BodyWriter.WriteAsync() work exactly the same
await g.Writer.WriteAsync(new ReadOnlyMemory<byte>(bytes));
await g.Writer.FlushAsync();
await Task.Delay(1000);
}
await g.CompleteAsync();
}
Variations I tried with and without DisableBufering() as well as writing to a pipe (IHttpResponseBodyFeature.Writer vs HttpContext.Response.Body) didn't seem to make a difference.
In curl it shows messages right away, however in Chrome and some rest clients it waits for the whole stream to show up.
So I would recommend testing your code behavior with a client that doesn't wait for the whole stream to present it. Another option I am still checking if aspnet core automatically picks up compression possibility if client asks for it even though compression is not configured in the pipeline.
So I would recomm
Reading Stephen Cleary take on not blocking on Async code I write something like this
public static async Task<JObject> GetJsonAsync(Uri uri)
{
using (var client = new HttpClient())
{
var jsonString = await client.GetStringAsync(uri).ConfigureAwait(false);
return JObject.Parse(jsonString);
}
}
public async void Button1_Click(...)
{
var json = await GetJsonAsync(...);
textBox1.Text=json;
}
so far so good, I understand that after the ConfigureAwait the method is going to continue running on a different context after GetStringAsync returns.
but what about if I want to use something like MessageBox (which is UI) like this
public static async Task<JObject> GetJsonAsync(Uri uri)
{
if(someValue<MAXVALUE)
{
using (var client = new HttpClient())
{
//var jsonString = await client.GetStringAsync(uri); //starts the REST request
var jsonString = await client.GetStringAsync(uri).ConfigureAwait(false);
return JObject.Parse(jsonString);
}
}
else
{
MessageBox.Show("The parameter someValue is too big!");
}
}
can I do this?
Even more complicated, how about this?
public static async Task<JObject> GetJsonAsync(Uri uri)
{
if(someValue<MAXVALUE)
{
try{
using (var client = new HttpClient())
{
//var jsonString = await client.GetStringAsync(uri); //starts the REST request
var jsonString = await client.GetStringAsync(uri).ConfigureAwait(false);
return JObject.Parse(jsonString);
}
}
catch(Exception ex)
{
MessageBox.Show("An Exception was raised!");
}
}
else
{
MessageBox.Show("The parameter someValue is too big!");
}
}
Can I do this?
Now, I am thinking perhaps all the message boxes should be called outside GetJsonAync as good design, but my question is can the above thing be done?
can I do this? [use a MessageBox]
Yes, but mainly because it has nothing to do with async/await or threading.
MessageBox.Show() is special, it is a static method and is documented as thread-safe.
You can show a MessageBox from any thread, any time.
So maybe it was the wrong example, but you do have MessageBox in the title.
public static async Task<JObject> GetJsonAsync(Uri uri)
{
try{
... // old context
... await client.GetStringAsync(uri).ConfigureAwait(false);
... // new context
}
catch
{
// this might bomb
someLabel.Text = "An Exception was raised!";
}
}
In this example, there could be code paths where the catch runs on the old and other paths where it runs on the new context.
Bottom line is: you don't know and should assume the worst case.
I would not use a Message Box, as it is very limited, and dated.
Also, Pop up's are annoying.
Use your own user control which enables user interaction the way you intend it.
In the context of Winforms / WPF / (and I guess UWP), only a single thread can manipulate the UI. Other threads can issue work to it via a queue of actions which eventually get invoked.
This architecture prevents other threads from constantly poking at the UI, which can make UX very janky (and thread unsafe).
The only way to communicate with it the UI work queue (in Winforms) is via the System.Windows.Form.Controls.BeginInvoke instance method, found on every form and control.
In your case:
public async void Button1_Click(...)
{
var json = await GetJsonAsync(...).ConfigureAwait(false);
BeginInvoke(UpdateTextBox, json);
}
private void UpdateTextBox(string value)
{
textBox1.Text=json;
}
I've been reading examples for a long time now, but unfortunately I've been unable to apply the solutions to the code I'm working with. Some quick Facts/Assorted Info:
1) I'm new to C#
2) The code posted below is modified from Amazon Web Services (mostly stock)
3) Purpose of code is to compare server info to offline already downloaded info and create a list of need to download files. This snip is for the list made from the server side, only option with AWS is to call async, but I need this to finish before moving forward.
public void InitiateSearch()
{
UnityInitializer.AttachToGameObject(this.gameObject);
//these are the access key and secret access key for credentials
BasicAWSCredentials credentials = new BasicAWSCredentials("secret key", "very secret key");
AmazonS3Config S3Config = new AmazonS3Config()
{
ServiceURL = ("url"),
RegionEndpoint = RegionEndpoint.blahblah
};
//Setting the client to be used in the call below
AmazonS3Client Client = new AmazonS3Client(credentials, S3Config);
var request = new ListObjectsRequest()
{
BucketName = "thebucket"
};
Client.ListObjectsAsync(request, (responseObject) =>
{
if (responseObject.Exception == null)
{
responseObject.Response.S3Objects.ForEach((o) =>
{
int StartCut = o.Key.IndexOf(SearchType) - 11;
if (SearchType == o.Key.Substring(o.Key.IndexOf(SearchType), SearchType.Length))
{
if (ZipCode == o.Key.Substring(StartCut + 12 + SearchType.Length, 5))
{
AWSFileList.Add(o.Key + ", " + o.LastModified);
}
}
}
);
}
else
{
Debug.Log(responseObject.Exception);
}
});
}
I have no idea how to apply await to the Client.ListObjectsAsync line, I'm hoping you all can give me some guidance and let me keep my hair for a few more years.
You can either mark your method async and await it, or you can call .Wait() or .Result() on the Task you're given back.
I have no idea how to apply await to the Client.ListObjectsAsync line
You probably just put await in front of it:
await Client.ListObjectsAsync(request, (responseObject) => ...
As soon as you do this, Visual Studio will give you an error. Take a good look at the error message, because it tells you exactly what to do next (mark InitiateSearch with async and change its return type to Task):
public async Task InitiateSearchAsync()
(it's also a good idea to add an Async suffix to follow the common pattern).
Next, you'd add an await everywhere that InitiateSearchAsync is called, and so on.
I'm assuming Client.ListObjectsAsync returns a Task object, so a solution for your specific problem would be this:
public async void InitiateSearch()
{
//code
var collection = await Client.ListObjectsAsync(request, (responseObject) =>
{
//code
});
foreach (var item in collection)
{
//do stuff with item
}
}
the variable result will now be filled with the objects. You may want to set the return type of InitiateSearch() to Task, so you can await it too.
await InitiateSearch(); //like this
If this method is an event handler of some sort (like called by the click of a button), then you can keep using void as return type.
A simple introduction from an unpublished part of the documentation for async-await:
Three things are needed to use async-await:
The Task object: This object is returned by a method which is executed asynchronous. It allows you to control the execution of the method.
The await keyword: "Awaits" a Task. Put this keyword before the Task to asynchronously wait for it to finish
The async keyword: All methods which use the await keyword have to be marked as async
A small example which demonstrates the usage of this keywords
public async Task DoStuffAsync()
{
var result = await DownloadFromWebpageAsync(); //calls method and waits till execution finished
var task = WriteTextAsync(#"temp.txt", result); //starts saving the string to a file, continues execution right await
Debug.Write("this is executed parallel with WriteTextAsync!"); //executed parallel with WriteTextAsync!
await task; //wait for WriteTextAsync to finish execution
}
private async Task<string> DownloadFromWebpageAsync()
{
using (var client = new WebClient())
{
return await client.DownloadStringTaskAsync(new Uri("http://stackoverflow.com"));
}
}
private async Task WriteTextAsync(string filePath, string text)
{
byte[] encodedText = Encoding.Unicode.GetBytes(text);
using (FileStream sourceStream = new FileStream(filePath, FileMode.Append))
{
await sourceStream.WriteAsync(encodedText, 0, encodedText.Length);
}
}
Some thing to note:
You can specify a return value from an asynchronous operations with Task. The await keyword waits till the execution of the method finishes, and returns the string.
the Task object contains the status of the execution of the method, it can be used as any other variable.
if an exception is thrown (for example by the WebClient) it bubbles up at the first time the await keyword is used (in this example at the line string result (...))
It is recommended to name methods which return the Task object as MethodNameAsync
For more information about this take a look at http://blog.stephencleary.com/2012/02/async-and-await.html.
I have a method that uploads one file to server. Right now is working besides any bad coding on the method (Im new to Task library).
Here is the code that uploads a file to server:
private async void UploadDocument()
{
var someTask = await Task.Run<bool>(() =>
{
// open input stream
using (System.IO.FileStream stream = new System.IO.FileStream(_cloudDocuments[0].FullName, System.IO.FileMode.Open, System.IO.FileAccess.Read))
{
using (StreamWithProgress uploadStreamWithProgress = new StreamWithProgress(stream))
{
uploadStreamWithProgress.ProgressChanged += uploadStreamWithProgress_ProgressChanged;
// start service client
SiiaSoft.Data.FieTransferWCF ws = new Data.FieTransferWCF();
// upload file
ws.UploadFile(_cloudDocuments[0].FileName, (long)_cloudDocuments[0].Size, uploadStreamWithProgress);
// close service client
ws.Close();
}
}
return true;
});
}
Then I have a ListBox where I can drag & drop multiple files so what I want to do is to do a FOR LOOP within the ListBox files and then call UploadDocument(); but I want to first upload 1st file in the listBox then when completed continue with the second file and so on...
Any clue on the best way to do it?
Thanks a lot.
You should make your UploadDocument return Task. Then you can await the task in a loop. For example:
private async Task UploadAllDocuments()
{
string[] documents = ...; // Fetch the document names
foreach (string document in documents)
{
await UploadDocument(document);
}
}
private async Task UploadDocument(string document)
{
// Code as before, but use document instead of _cloudDocuments[0]
}
In fact, your UploadDocument can be made simpler anyway:
private Task UploadDocument()
{
return Task.Run<bool>(() =>
{
// Code as before
});
}
Wrapping that in an async method isn't particularly useful.
(You may well want to change the type to not be string - it's not clear what _cloudDocuments is.)
In general, you should always make an async method return Task or Task<T> unless you have to make it return void to comply with an event-handling pattern.