Develop Reference

winforms progress bar not updating properly according to Google Cloud IUploadProgress c#

So I'm uploading objects to Google Cloud Storage with UploadObjectAsync. I update a progress bar using IUploadProgress.Status but the progress bar doesn't seem to be syncing correctly with the Uploading Status. Basically when the upload is complete I change a label's text saying "Upload completed". That happens when the upload is complete but sometimes the progress bar is still empty (and fills up afterwards). I can't figure out if the Completed status happens to fast for the progress bar to update.
public async void UploadAsync(string projectID, string bucketName, GoogleCredential credentials, string filePath,string _objectName, StringBuilder _objectData, List<string> _metaData, string objectName = null)
{
// Instantiates a client.
using (var storageClient = Google.Cloud.Storage.V1.StorageClient.Create(credentials))
{
try
{
string[] lines = _objectData.ToString().Split(new[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries);
using (var memStream = new MemoryStream())
using (var memStream2 = new MemoryStream())
using (var streamWriter = new StreamWriter(memStream))
using (var streamWriter2 = new StreamWriter(memStream2))
using (var csvWriter = new CsvWriter(streamWriter, CultureInfo.InvariantCulture) )
{
//for data.csv
foreach (var line in lines)
{
string[] lineItems = line.Split(',');
foreach (var singleItem in lineItems) {
csvWriter.WriteField(singleItem);
}
csvWriter.NextRecord();
}
streamWriter.Flush(); //always flush streamWriter after writing
memStream.Position = 0;
foreach (var entry in _metaData)
{
streamWriter2.WriteLine(entry);
}
streamWriter2.Flush();
memStream2.Position = 0;
using (FileStream fileMeta = new FileStream(_objectName + "meta.txt", FileMode.Create, FileAccess.Write))
{
memStream2.CopyTo(fileMeta);
}
setMax((int)(memStream.Length + memStream2.Length)); //sets progress bar max to be length of both memory streams
// set minimum chunksize just to see progress updating
var uploadObjectOptions = new Google.Cloud.Storage.V1.UploadObjectOptions
{
ChunkSize = Google.Cloud.Storage.V1.UploadObjectOptions.MinimumChunkSize
};
// Hook up the progress callback
var progressReporter = new Progress<Google.Apis.Upload.IUploadProgress>(OnUploadProgress);
Console.WriteLine(bucketName);
Console.WriteLine("objectName" + objectName);
bucketName = "csv-upload-bucket";
/*storageClient.UploadObject*/
await storageClient.UploadObjectAsync(
bucketName,
_objectName + ".csv",
"application/csv",
memStream,
uploadObjectOptions,
progress: progressReporter).ConfigureAwait(true);
await storageClient.UploadObjectAsync(
bucketName,
_objectName + "meta.txt",
"application/text",
memStream2,
uploadObjectOptions,
progress: progressReporter).ConfigureAwait(true);
}
}
catch (Google.GoogleApiException e)
when (e.Error.Code == 409)
{
// When creating the bucket - The bucket already exists. That's fine.
Console.WriteLine(e.Error.Message);
}
catch (Exception e)
{
// other exception
Console.WriteLine(e.Message);
}
}
}
void OnUploadProgress(Google.Apis.Upload.IUploadProgress progress)
{
switch (progress.Status)
{
case Google.Apis.Upload.UploadStatus.Starting:
setMin(0); //sets progress bar value to 0
UpdateProgressBar(0);
break;
case Google.Apis.Upload.UploadStatus.Completed:
UpdateProgressBar(getMax()); //sets progress bar value to max
setResultLabelText("Upload completed");
setLoadCompleteBttnEnabled(true); //enables OK bttn
setCancelLoadBttnEnabled(false);
break;
case Google.Apis.Upload.UploadStatus.Uploading:
UpdateProgressBar((int)progress.BytesSent);
setLoadCompleteBttnEnabled(false);
setCancelLoadBttnEnabled(true);
Console.WriteLine("busy uploading"); //this never seems to be called
break;
case Google.Apis.Upload.UploadStatus.Failed:
setResultLabelText("Upload failed");
Console.WriteLine(Environment.NewLine + progress.Exception);
break;
}
}

Send Multiple Posts At A Time

I am using a SQL Query to select data from my database and storing the returned results in a Dictionary<string, string> I then use a foreach loop and Server.UrlEncode to create a Querystring like such
foreach (var onetwo in privatedictionary)
dataToPost = onetwo.Key+"="+Server.UrlEncode(onetwo.Value)+"&";
then once the data has been compiled into dataToPost I use HttpWebRequest to send the data
HttpWebRequest wbrq = (HttpWebRequest)WebRequest.Create("www.sitetohit.com");
{
using (StreamWriter sw = new StreamWriter(wbrq.GetRequestStream()))
{
sw.Write(dataToPost);
}
}
What is a way using Visual Studio 2015 to send multiple Posts at once?

There are multiple ways, the best that I experienced are bellows:
Parallel.ForEach(privatedictionary, (dataToPost) =>
{
HttpWebRequest wbrq = (HttpWebRequest)WebRequest.Create("www.sitetohit.com");
{
using (StreamWriter sw = new StreamWriter(wbrq.GetRequestStream()))
{
sw.Write(dataToPost);
}
}
});
Or:
foreach (var dataToPost in privatedictionary)
{
Task.Run(async () =>
{
Foo foo = new Foo();
await foo.BarAsync(dataToPost);
});
}
//Asynchronous Handling
public class Foo
{
public async Task BarAsync(string dataToPost)
{
HttpWebRequest wbrq = (HttpWebRequest)WebRequest.Create("www.sitetohit.com");
{
using (StreamWriter sw = new StreamWriter(wbrq.GetRequestStream()))
{
await sw.WriteAsync(dataToPost);
}
}
}
}
You can use this:
Parallel.ForEach(privatedictionary, (onetwo) =>
{
var dataToPost = onetwo.Key + "=" + Server.UrlEncode(onetwo.Value) + "&";
HttpWebRequest wbrq = (HttpWebRequest)WebRequest.Create("www.sitetohit.com");
{
using (StreamWriter sw = new StreamWriter(wbrq.GetRequestStream()))
{
sw.Write(dataToPost);
}
}
});

Multithreading for making http post requests to web service

I want to send multiple HTTP post requests to a Web Service in C# .For example , if n=3 then http post requests from 3 xml files should be made and also the response should be written in a file.Once the first 3 requests are made then the next 3 requests will be made .
So i made the following code but i was getting random outputs at first. But now i am getting either out of index range exception in the inner for loop or Internal server error (500). Plz suggest appropriate changes. I am using .NET4.0
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Threading;
using System.Xml;
using System.Net;
using System.Threading.Tasks;
namespace ConsoleApplication5
{
class Program
{
static void Main(string[] args)
{
int n = 0;
Console.WriteLine("Enter the number");
string s = Console.ReadLine();
int.TryParse(s, out n);
string path = "C:\\";
string[] files = null;
files = Directory.GetFiles(path, "*.xml", SearchOption.TopDirectoryOnly);
List<Task> tasks = new List<Task>(files.Length);
for (int i = 0; i < files.Length; i += n)
{
for (int j = 0; j < n; j++)
{
int x = i + j;
if (x < files.Length && files[x] != null)
{
Task t = new Task(() => function(files[x]));
t.Start();
tasks.Add(t);
}
}
if (tasks.Count > 0)
{
Task.WaitAll(tasks.ToArray(), Timeout.Infinite); // or less than infinite
tasks.Clear();
}
}
}
public static void function(string temp)
{
XmlDocument doc = new XmlDocument();
doc.Load(temp);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://10.76.22.135/wpaADws/ADService.asmx");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
Stream stream = request.GetRequestStream();
doc.Save(stream);
stream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader rd = new StreamReader(response.GetResponseStream()))
{
string soapResult = rd.ReadToEnd();
doc.LoadXml(soapResult);
File.WriteAllText(temp, doc.DocumentElement.InnerText);
//XmlTextWriter xml=new XmlTextWriter(
Console.WriteLine(soapResult);
Console.ReadKey();
}
}
}
}

This code works .
Explaination :
Firstly the user gives the source and destination paths for the .xml files.
Directory.getFiles() helps us to get the .xml files in the string array .
(we have to pass .xml as a parameter) .
SO now what basically happens is for each file we get at the source pat , a thread is created .
But say if the user wants to send "n" requests at a time , then n threads are created at a time.
And the next set of threads are not created unless the previous threads are finished executing.
This is ensured by thread.Join().
And after a request is made to the web service , we get the response by getResponse() and the response is written in .xml files which are stored at the destination paths.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Threading;
using System.Xml;
using System.Net;
namespace ConsoleApplication4
{
class Program
{
int flag = 1;
string destination;
string source;
static void Main(string[] args)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.WriteLine("**************************** Send HTTP Post Requests **************************");
int n = 0;
Program p = new Program();
Console.WriteLine("Enter the number of requests you want to send at a time");
string s = Console.ReadLine();
int.TryParse(s, out n);
Console.WriteLine("Enter Source");
p.source = Console.ReadLine();
Console.WriteLine("Enter Destination");
p.destination = Console.ReadLine();
string[] files = null;
files = Directory.GetFiles(p.source, "*.xml", SearchOption.TopDirectoryOnly);
Thread[] thread = new Thread[files.Length];
int len = files.Length;
for (int i = 0; i<len; i+=n)
{
int x = i;
//Thread.Sleep(5000);
for (int j = 0; j < n && x < len; j++)
{
var localx = x;
thread[x] = new Thread(() => function(files[localx], p));
thread[x].Start();
Thread.Sleep(50);
//thread[x].Join();
x++;
}
int y = x - n;
for (; y < x; y++)
{
int t = y;
thread[t].Join();
}
}
// thread[0] = new Thread(() => function(files[0]));
//thread[0].Start();
Console.ReadKey();
}
public static void function(string temp,Program p)
{
XmlDocument doc = new XmlDocument();
doc.Load(temp);
string final_d=p.destination + "response " + p.flag + ".xml";
p.flag++;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://10.76.22.135/wpaADws/ADService.asmx");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
Stream stream = request.GetRequestStream();
doc.Save(stream);
stream.Close();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
using (StreamReader rd = new StreamReader(response.GetResponseStream()))
{
string soapResult = rd.ReadToEnd();
doc.LoadXml(soapResult);
File.WriteAllText(final_d, doc.DocumentElement.InnerText);
//XmlTextWriter xml=new XmlTextWriter(
Console.WriteLine(soapResult);
//Console.ReadKey();
}
}
}
}

The IndexOutOfRangeException you experienced in your original post was due to the improper index handling on the last batch of files you were processing. That last batch can be incomplete and you treated that as a regular batch of set size
(n=3 in your post)
Since you're moving to TPL and Tasks, I suggest Parallel Programming with Microsoft .NET, and the pipeline pattern which seems very appropriate to your scenario. You can harness the power of concurrent collections and the producer/consumer pattern together with the pipeline, like below. BlockingCollection ensures concurrent adding of items and the BlockingCollection.GetConsumingEnumerable call produces a consuming blocking enumerator for your collection.
const int BUFFER_SIZE = 3; // no concurrent items to process
const string XML_FOLDER_PATH = "<whatever>";
public static void Pipeline()
{
var bufferXmlFileNames = new BlockingCollection<string>(BUFFER_SIZE);
var bufferInputXmlDocuments = new BlockingCollection<XmlDocument>(BUFFER_SIZE);
var bufferWebRequests = new BlockingCollection<HttpWebRequest>(BUFFER_SIZE);
var bufferSoapResults = new BlockingCollection<string>(BUFFER_SIZE);
var f = new TaskFactory(TaskCreationOptions.LongRunning, TaskContinuationOptions.None);
// Stage 1: get xml file paths
var stage1 = f.StartNew(() => {
try
{
foreach (var phrase in Directory.GetFiles(XML_FOLDER_PATH, "*.xml", SearchOption.TopDirectoryOnly))
{ // build concurrent collection
bufferXmlFileNames.Add(phrase);
}
}
finally
{ // no more additions acceptedin
bufferXmlFileNames.CompleteAdding();
}
});
// Stage 2: ProduceInputXmlDocuments(bufferXmlFileNames, bufferInputXmlDocuments)
var stage2 = f.StartNew(() => {
try
{
foreach (var xmlFileName in bufferXmlFileNames.GetConsumingEnumerable())
{
XmlDocument doc = new XmlDocument();
doc.Load(xmlFileName);
bufferInputXmlDocuments.Add(doc);
}
}
finally
{
bufferInputXmlDocuments.CompleteAdding();
}
});
// Stage 3: PostRequests(BlockingCollection<XmlDocument> xmlDocs, BlockingCollection<HttpWebRequest> posts)
var stage3 = f.StartNew(() => {
try
{
foreach (var xmlDoc in bufferInputXmlDocuments.GetConsumingEnumerable())
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://10.76.22.135/wpaADws/ADService.asmx");
request.ContentType = "text/xml;charset=\"utf-8\"";
request.Accept = "text/xml";
request.Method = "POST";
//
Stream stream = request.GetRequestStream();
xmlDoc.Save(stream);
stream.Close();
//
bufferWebRequests.Add(request);
}
}
finally
{
bufferWebRequests.CompleteAdding();
}
});
// Stage 4: ProcessResponses(bufferWebRequests, bufferSoapResults)
var stage4 = f.StartNew(() =>
{
try
{
foreach (var postRequest in bufferWebRequests.GetConsumingEnumerable())
{
HttpWebResponse response = (HttpWebResponse)postRequest.GetResponse();
using (StreamReader rd = new StreamReader(response.GetResponseStream()))
{
string soapResult = rd.ReadToEnd();
bufferSoapResults.Add(soapResult);
}
}
}
finally
{
bufferSoapResults.CompleteAdding();
}
});
// stage 5: update UI
var stage5 = f.StartNew(() =>
{
foreach (var soapResult in bufferSoapResults.GetConsumingEnumerable())
{
Console.WriteLine(soapResult);
}
});
// display blocking collection load state,
// the number of elements in each blocking collection of the pipeline stages
// you can supress this call completely, because it is informational only
var stageDisplay = f.StartNew(
() =>
{
while (true)
{
Console.WriteLine("{0,10} {1,10} {2,10} {3,10}", bufferXmlFileNames.Count, bufferInputXmlDocuments.Count, bufferWebRequests.Count, bufferSoapResults.Count);
//check last stage completion
if (stage5.IsCompleted)
return;
}
}
);
Task.WaitAll(stage1, stage2, stage3, stage4, stage5); //or
//Task.WaitAll(stage1, stage2, stage3, stage4, stage5, stageDisplay);
}

How about using tasks like this:
List<Task> tasks = new List<Task>(n);
for (int i = 0; i < files.Length; i += n)
{
for (int j = 0; j < n; j++)
{
int x = i + j;
if (x < files.Length && files[x] != null)
{
Task t = new Task(() => function(files[x]));
t.Start();
tasks.Add(t);
}
}
if (tasks.Count > 0)
{
Task.WaitAll(tasks.ToArray(), Timeout.Infinite); // or less than infinite
tasks.Clear();
}
}
I tried to be a little tidier on the indexing...
Also, note that the int x = i + j; in the inner loop is important due to how C# captures variables for the lambda.
If the problem is tracing down indexing arithmetic, maybe use indexing variables with meaningful names?
List<Task> tasks = new List<Task>(taskCount);
for (int filesIdx = 0; filesIdx < files.Length; filesIdx += taskCount)
{
for (int tasksIdx = 0; tasksIdx < taskCount; tasksIdx++)
{
int index = filesIdx + tasksIdx;
if (index < files.Length && files[index] != null)
{
Task task = new Task(() => function(files[index]));
task.Start();
tasks.Add(task);
}
}
if (tasks.Count > 0)
{
Task.WaitAll(tasks.ToArray(), Timeout.Infinite); // or less than infinite
tasks.Clear();
}
}

Is my PLINQ code threadsafe?

I have the following code:
static void Main(string[] args)
{
DateTime currentDay = DateTime.Now;
List<Task> taskList = new List<Task>();
List<string> markets = new List<string>() { "amex", "nasdaq", "nyse", "global" };
Parallel.ForEach(markets, market =>
{
Downloads.startInitialMarketSymbolsDownload(market);
}
);
Console.WriteLine("All downloads finished!");
}
public static void startInitialMarketSymbolsDownload(string market)
{
try
{
object valueTypeLock = new object();
List<string> symbolList = new List<string>() { "GOOG", "YHOO", "AAP" }
var historicalGroups = symbolList.AsParallel().Select((x, i) => new { x, i })
.GroupBy(x => x.i / 100)
.Select(g => g.Select(x => x.x).ToArray());
historicalGroups.AsParallel().ForAll(g => {
lock (valueTypeLock) {
getHistoricalStockData(g, market);
}
});
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
Console.WriteLine(ex.StackTrace);
}
}
public static void getHistoricalStockData(string[] symbols, string market)
{
// download data for list of symbols and then upload to db tables
Uri uri;
string url, line;
decimal open = 0, high = 0, low = 0, close = 0, adjClose = 0;
DateTime date;
Int64 volume = 0;
string[] lineArray;
List<string> symbolError = new List<string>();
Dictionary<string, string> badNameError = new Dictionary<string, string>();
System.Net.ServicePointManager.DefaultConnectionLimit = 1000;
Parallel.ForEach(symbols, symbol =>
{
url = "http://ichart.finance.yahoo.com/table.csv?s=" + symbol + "&a=00&b=1&c=1900&d=" + (DateTime.Now.Month - 1) + "&e=" + DateTime.Now.Day + "&f=" + DateTime.Now.Year + "&g=d&ignore=.csv";
uri = new Uri(url);
using (ooplesfinanceEntities entity = new ooplesfinanceEntities())
using (WebClient client = new WebClient())
using (Stream stream = client.OpenRead(uri))
using (StreamReader reader = new StreamReader(stream))
{
entity.Database.Connection.Open();
while (reader.EndOfStream == false)
{
line = reader.ReadLine();
lineArray = line.Split(',');
// if it isn't the very first line
if (lineArray[0] != "Date")
{
switch (market)
{
case "amex":
DailyAmexData amexData = new DailyAmexData();
var amexQuery = from r in entity.DailyAmexDatas.AsParallel().AsEnumerable()
where r.Date == date
select new StockData { Close = r.AdjustedClose };
List<StockData> amexResult = amexQuery.AsParallel().ToList();
if (amexResult.AsParallel().Count() > 0) // **never hits this breakpoint line**
{
// add the row to the table if it isn't there
// now checks for stock splits and updates if necessary
if (amexResult.AsParallel().FirstOrDefault().Close != adjClose)
{
// this means there is a stock split so it needs to have the other adjusted close prices updated
amexResult.AsParallel().FirstOrDefault().Close = adjClose;
}
else
{
continue;
}
}
else
{
// set the data then add it
amexData.Symbol = symbol;
entity.DailyAmexDatas.Add(amexData);
}
break;
default:
break;
}
}
}
// now save everything
entity.SaveChanges();
Console.WriteLine(symbol + " added to the " + market + " database!");
}
}
);
}
Everything starts fine and I get no exceptions but I'm getting no results and the code gets stuck one the line that I marked and the memory keeps shooting up. I figured the problem was with the above code because maybe I was doing something wrong. I just don't know where to start as this is my first time dealing with plinq/parallel processing and the tutorials don't show anything this complex.

The process cannot access the file because it is being used by another process. XML

I am calling below method in a loop with same xmlRequestPath and xmlResponsePath files. Two loop counts it executes fine in the 3rd iteration I am getting exception "The process cannot access the file because it is being used by another process.".
public static void UpdateBatchID(String xmlRequestPath, String xmlResponsePath)
{
String batchId = "";
XDocument requestDoc = null;
XDocument responseDoc = null;
lock (locker)
{
using (var sr = new StreamReader(xmlRequestPath))
{
requestDoc = XDocument.Load(sr);
var element = requestDoc.Root;
batchId = element.Attribute("BatchID").Value;
if (batchId.Length >= 16)
{
batchId = batchId.Remove(0, 16).Insert(0, DateTime.Now.ToString("yyyyMMddHHmmssff"));
}
else if (batchId != "") { batchId = DateTime.Now.ToString("yyyyMMddHHmmssff"); }
element.SetAttributeValue("BatchID", batchId);
}
using (var sw = new StreamWriter(xmlRequestPath))
{
requestDoc.Save(sw);
}
using (var sr = new StreamReader(xmlResponsePath))
{
responseDoc = XDocument.Load(sr);
var elementResponse = responseDoc.Root;
elementResponse.SetAttributeValue("BatchID", batchId);
}
using (var sw = new StreamWriter(xmlResponsePath))
{
responseDoc.Save(sw);
}
}
Thread.Sleep(500);
requestDoc = null;
responseDoc = null;
}
Exception is occurring at using (var sw = new StreamWriter(xmlResponsePath)) in above code.
Exception:
The process cannot access the file 'D:\Projects\ESELServer20130902\trunk\Testing\ESL Server Testing\ESLServerTesting\ESLServerTesting\TestData\Assign\Expected Response\Assign5kMACResponse.xml' because it is being used by another process.

Maybe at the third loop the stream is still being closed, so it tells you that it is non accessible. Try waiting a bit before calling it again in the loop, for example:
while (...)
{
UpdateBatchID(xmlRequestPath, xmlResponsePath);
System.Threading.Thread.Sleep(500);
}
Or, close explicitly the stream instead of leaving the work to the garbage collector:
var sr = new StreamReader(xmlResponsePath);
responseDoc = XDocument.Load(sr);
....
sr.Close();

Instead of using two streams, a Write and a Read stream, try using only a FileStream, since the problem might be that after loading the file the stream remains opened until the garbadge collector actives.
using (FileSteam f = new FileStream(xmlResponsePath))
{
responseDoc = XDocument.Load(sr);
var elementResponse = responseDoc.Root;
elementResponse.SetAttributeValue("BatchID", batchId);
responseDoc.Save(sw);
}