The application I am building allows a user to upload a .csv file, which will ultimately fill in fields of an existing SQL table where the Ids match. First, I am using LinqToCsv and a foreach loop to import the .csv into a temporary table. Then I have another foreach loop where I am trying to loop the rows from the temporary table into an existing table where the Ids match.
Controller Action to complete this process:
[HttpPost]
public ActionResult UploadValidationTable(HttpPostedFileBase csvFile)
{
var inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
var cc = new CsvContext();
var filePath = uploadFile(csvFile.InputStream);
var model = cc.Read<Credit>(filePath, inputFileDescription);
try
{
var entity = new TestEntities();
var tc = new TemporaryCsvUpload();
foreach (var item in model)
{
tc.Id = item.Id;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
tc.CreditDeniedDate = item.CreditDeniedDate;
tc.CreditDeniedReasonId = item.CreditDeniedReasonId;
tc.CreditDeniedNotes = item.CreditDeniedNotes;
entity.TemporaryCsvUploads.Add(tc);
}
var idMatches = entity.Authorizations.ToList().Where(x => x.Id == tc.Id);
foreach (var number in idMatches)
{
number.CreditInvoiceDate = tc.CreditInvoiceDate;
number.CreditInvoiceNumber = tc.CreditInvoiceNumber;
number.CreditInvoiceAmount = tc.CreditInvoiceAmount;
number.CreditDeniedDate = tc.CreditDeniedDate;
number.CreditDeniedReasonId = tc.CreditDeniedReasonId;
number.CreditDeniedNotes = tc.CreditDeniedNotes;
}
entity.SaveChanges();
entity.Database.ExecuteSqlCommand("TRUNCATE TABLE TemporaryCsvUpload");
TempData["Success"] = "Updated Successfully";
}
catch (LINQtoCSVException)
{
TempData["Error"] = "Upload Error: Ensure you have the correct header fields and that the file is of .csv format.";
}
return View("Upload");
}
The issue in the above code is that tc is inside the first loop, but the matches are defined after the loop with var idMatches = entity.Authorizations.ToList().Where(x => x.Id == tc.Id);, so I am only getting the last item of the first loop.
So I would need to put var idMatches = entity.Authorizations.ToList().Where(x => x.Id == tc.Id); in the first loop, but then I can't access it in the second. If I nest the second loop then it is way to slow. Is there any way I could put the above statement in the first loop and still access it. Or any other ideas to accomplish the same thing? Thanks!
Instead of using multiple loops, keep track of processed IDs as you go and then exclude any duplicates.
[HttpPost]
public ActionResult UploadValidationTable(HttpPostedFileBase csvFile)
{
var inputFileDescription = new CsvFileDescription
{
SeparatorChar = ',',
FirstLineHasColumnNames = true
};
var cc = new CsvContext();
var filePath = uploadFile(csvFile.InputStream);
var model = cc.Read<Credit>(filePath, inputFileDescription);
try
{
var entity = new TestEntities();
var tcIdFound = new HashSet<string>();
foreach (var item in model)
{
if (tcIdFound.Contains(item.Id))
{
continue;
}
var tc = new TemporaryCsvUpload();
tc.Id = item.Id;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
tc.CreditDeniedDate = item.CreditDeniedDate;
tc.CreditDeniedReasonId = item.CreditDeniedReasonId;
tc.CreditDeniedNotes = item.CreditDeniedNotes;
entity.TemporaryCsvUploads.Add(tc);
}
entity.SaveChanges();
entity.Database.ExecuteSqlCommand("TRUNCATE TABLE TemporaryCsvUpload");
TempData["Success"] = "Updated Successfully";
}
catch (LINQtoCSVException)
{
TempData["Error"] = "Upload Error: Ensure you have the correct header fields and that the file is of .csv format.";
}
return View("Upload");
}
If you want to make sure you get the last value for any duplicate ids, then store each TemporaryCsvUpload record in a dictionary instead of using only a HashSet. Same basic idea though.
Declare idMatches before the first loop, but don't instantiate it or set its value to null. Then you'll be able to use it inside both loops. After moving the declaration before the first loop, you'll still end up having the values from the last iteration using a simple Where. You'll need to concatenate the already existing list with results for the current iteration.
Related
I am trying to save a large cvs file into the database. The file i am using is about 7000 rows and each row contains 14 columns. I have to generate and tag each column of every row with a topic id i pass in my api. After saving each item i then loop through the actual data and i use the generated id to save each data in another table. My problem is i have nested foreach loops and in the first loop i call db.saveChanges() after taking each column in every row so i can reference the generated id. but that is A LOT of saveChanges() calls that are made before processing the data.
For an example:
public static void Save(TopicRequest req){
using(var db = new DbContext()){
foreach(var row in req.items){
var obj = new Entity{
topicId = req.topicId,
year = req.year
};
db.Add(obj);
db.saveChanges();
foreach(var col in row){
var newData = new Entity{
TopicObjId = obj.id,
Value = col
}
db.TopicData.Add(newData);
}
db.saveChanges();
}
}
}
so for a 7000 row file with 14 columns that means that my first loop will make a call to save into the db 98,000 times. This is causing a timeout and the file saved. How can i probably handle such large amounts of data in this way.
I suggest to use AddRange to improve the performance.
Add vs AddRange
Here's an example:
public async Task Save(TopicRequest req)
{
using(var db = new DbContext())
{
var list1 = new List<Entity1>();
var list2 = new List<Entity2>();
foreach(var row in req.items)
{
var obj = new Entity1
{
topicId = req.topicId,
year = req.year
};
list1.Add(obj);
}
db.Topic.AddRange(list1);
await db.SaveChangesAsync();
// this may not be necessary
await db.Entry(list1).ReloadAsync():
foreach(var obj in list1)
{
var newData = new Entity2
{
TopicObjId = obj.topicId,
Value = obj.value
};
list2.Add(newData);
}
db.TopicData.AddRange(list2);
await db.SaveChangesAsync();
}
}
I have a database called ebookstore.db as below:
and JSON as below:
I want when slug on JSON is not the same as a title in the database, it will display the amount of data with a slug on JSON which is not same as a title in the database in ukomikText.
Code:
string judulbuku;
try
{
string urlPath1 = "https://...";
var httpClient1 = new HttpClient(new HttpClientHandler());
httpClient1.DefaultRequestHeaders.TryAddWithoutValidation("KIAT-API-KEY", "....");
var values1 = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string, string>("halaman", 1),
new KeyValuePair<string, string>("limit", 100),
};
var response1 = await httpClient1.PostAsync(urlPath1, new FormUrlEncodedContent(values1));
response1.EnsureSuccessStatusCode();
if (!response1.IsSuccessStatusCode)
{
MessageDialog messageDialog = new MessageDialog("Memeriksa update Komik gagal", "Gangguan Server");
await messageDialog.ShowAsync();
}
string jsonText1 = await response1.Content.ReadAsStringAsync();
JsonObject jsonObject1 = JsonObject.Parse(jsonText1);
JsonArray jsonData1 = jsonObject1["data"].GetArray();
foreach (JsonValue groupValue in jsonData1)
{
JsonObject groupObject = groupValue.GetObject();
string id = groupObject["id"].GetString();
string judul = groupObject["judul"].GetString();
string slug = groupObject["slug"].GetString();
BukuUpdate file1 = new BukuUpdate();
file1.ID = id;
file1.Judul = judul;
file1.Slug = slug;
List<String> title = sqlhelp.GetKomikData();
foreach (string juduldb in title)
{
judulbuku = juduldb.Substring(juduldb.IndexOf('.') + 1);
if (judulbuku != file1.Slug.Replace("-", "_") + ".pdf")
{
BukuData.Add(file1);
ListBuku.ItemsSource = BukuData;
}
else
{
ukomikText.Text = "belum tersedia komik yang baru";
ukomikText.Visibility = Visibility.Visible;
}
}
}
if (ListBuku.Items.Count > 0)
{
ukomikText.Text = BukuData.Count + " komik baru";
ukomikText.Visibility = Visibility.Visible;
jumlahbuku = BukuData.Count;
}
else
{
ukomikText.Text = "belum tersedia komik yang baru";
ukomikText.Visibility = Visibility.Visible;
}
public static List<String> GetKomikData()
{
List<String> entries = new List<string>();
using (SqliteConnection db =
new SqliteConnection("Filename=ebookstore.db"))
{
db.Open();
SqliteCommand selectCommand = new SqliteCommand
("SELECT title FROM books where folder_id = 67", db);
SqliteDataReader query = selectCommand.ExecuteReader();
while (query.Read())
{
entries.Add(query.GetString(0));
}
db.Close();
}
return entries;
}
BukuUpdate.cs:
public string ID { get; set; }
public string Judul { get; set; }
public string Slug { get; set; }
I have a problem, that is when checking slugs on JSON, then the slug that is displayed is the first slug is displayed repeatedly as much data in the database, after that show the second slug repeatedly as much data on the database, and so on, as below:
How to solve it so that slug on JSON is not displayed repeatedly (according to the amount of data on JSON)?
The problem is that you have two nested foreach loops. What the code does in simplified pseudocode:
For each item in JSON
Load all rows from DB
And for each loaded row
Check if the current JSON item matches the row from DB and if not, output
As you can see, if you have N items in the JSON and M rows in the database, this inevitably leads to N*M lines of output except for those rare ones where the JSON item matches a specific row in database.
If I understand it correctly, I assume that you instead want to check if there is a row that matches the JSON item and if not, output it. You could do this the following way:
List<String> title = sqlhelp.GetKomikData();
HashSet<string> dbItems = new HashSet<string>();
foreach (string juduldb in title)
{
judulbuku = juduldb.Substring(juduldb.IndexOf('.') + 1);
dbItems.Add( judulbuku );
}
...
foreach ( JsonValue groupValue in jsonData1 )
{
...
//instead of the second foreach
if ( !dbItems.Contains( file1.Slug.Replace("-", "_") + ".pdf" ) )
{
//item is not in database
}
else
{
//item is in database
}
}
Additional tips
Avoid calling GetKomikData inside the foreach. This method does not have any arguments and that means you are just accessing the database again and again without a reason, which takes time and slows down the execution significantly. Instead, call GetKomikData only once before the first foreach and then just use title variable.
Don't assign ItemsSource every time the collection changes. This will unnecessarily slow down the UI thread, as it will have to reload all the items with each loop. Instead, assign the property only once after the outer foreach
write your code in one language. When you start mixing variable names in English with Indonesian, the code becomes confusing and less readable and adds cognitive overhead.
avoid non-descriptive variable names like file1 or jsonObject1. The variable name should be clear and tell you what it contains. When there is a number at the end, it usually means it could be named more clearly.
use plurals for list variable names - instead of title use titles
I created code to load definitions from an external API. The code iterates through a list of words, looks up a definition for each and then I thought to use EF to insert these into my SQL Server database.
However if I run this twice it will load the same definitions the second time. Is there a way that I could make it so that EF does not add the row if it already exists?
public IHttpActionResult LoadDefinitions()
{
var words = db.Words
.AsNoTracking()
.ToList();
foreach (var word in words)
{
HttpResponse<string> response = Unirest.get("https://wordsapiv1.p.mashape.com/words/" + word)
.header("X-Mashape-Key", "xxxx")
.header("Accept", "application/json")
.asJson<string>();
RootObject rootObject = JsonConvert.DeserializeObject<RootObject>(response.Body);
var results = rootObject.results;
foreach (var result in results)
{
var definition = new WordDefinition()
{
WordId = word.WordId,
Definition = result.definition
};
db.WordDefinitions.Add(definition);
}
db.SaveChanges();
}
return Ok();
}
Also would appreciate if anyone has any suggestions as to how I could better implement this loading.
foreach (var result in results)
{
if(!(from d in db.WordDefinitions where d.Definition == result.definition select d).Any())
{
var definition = new WordDefinition()
{
WordId = word.WordId,
Definition = result.definition
};
db.WordDefinitions.Add(definition);
}
}
You can search for Definition value.
var wd = db.WordDefinition.FirstOrDefault(x => x.Definition == result.definition);
if(wd == null) {
var definition = new WordDefinition() {
WordId = word.WordId,
Definition = result.definition
};
db.WordDefinitions.Add(definition);
}
In this way you can get a WordDefinition that already have your value.
If you can also use WordId in the same way:
var wd = db.WordDefinition.FirstOrDefault(x => x.WordId == word.WordId);
I have two tables in an SQL database. They both have five fields: ID (PK), Number, InvoiceDate, InvoiceNumber, and InvoiceAmount. I am attempting to use Entity Framework to insert the InvoiceDate, InvoiceNumber, and InvoiceAmount where the field Number matches from one table to the other.
The context for table one is:
var tc = new TemporaryCsvUpload();
Table two:
var pt = new PermanentTestTable();
First, I inserted values into table 1 from a CSV, now I am trying to insert into table two where the Number field matches.
var entity = new CsvDbEntities1();
foreach (var item in model)
{
var tc = new TemporaryCsvUpload();
tc.Number = item.Number;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
entity.TemporaryCsvUploads.Add(tc);
entity.SaveChanges();
}
I am new to EF and any help would be appreciated. Thanks!
Sorry if I dont completely understand but here it goes based of this code :
foreach (var item in model)
{
var tc = new TemporaryCsvUpload();
tc.Number = item.Number;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
entity.TemporaryCsvUploads.Add(tc);
entity.SaveChanges();
}
After you have saved the first table now you query the second for the same record as :
var table2entity = entity.PermanentTestTable.Where(x => x.Number == tc.Number).Select(x => x).First();
This will query the second table and grab whatever entity is there with the same number
So your end code might look like :
var entity = new CsvDbEntities1();
foreach (var item in model)
{
var tc = new TemporaryCsvUpload();
tc.Number = item.Number;
tc.CreditInvoiceAmount = item.CreditInvoiceAmount;
tc.CreditInvoiceDate = item.CreditInvoiceDate;
tc.CreditInvoiceNumber = item.CreditInvoiceNumber;
entity.TemporaryCsvUploads.Add(tc);
entity.SaveChanges();
var table2entity = entity.PermanentTestTable.ToList();
table2entity = table2entity.Where(x => x.Number == tc.Number).Select(x => x).First();
table2entity.CreditInvoiceAmount = item.CreditInvoiceAmount;
//More values inserted here
entity.SaveChanges()
}
I ended up having to nest a foreach loop to iterate through every row with a Number field. I'm sure there is a better way to do this, but I am just happy it's working:
var entity = new CsvDbEntities1();
foreach (var item in model)
{
var tc = new TemporaryCsvUpload
{
PoNumber = item.Number,
CreditInvoiceAmount = item.CreditInvoiceAmount,
CreditInvoiceDate = item.CreditInvoiceDate,
CreditInvoiceNumber = item.CreditInvoiceNumber
};
entity.TemporaryCsvUploads.Add(tc);
var ptt = entity.PermanentTestTables.ToList().Where(x => x.Number == tc.Number);
foreach (var row in ptt)
{
row.CreditInvoiceDate = tc.CreditInvoiceDate;
row.CreditInvoiceNumber = tc.CreditInvoiceNumber;
row.CreditInvoiceAmount = tc.CreditInvoiceAmount;
}
entity.SaveChanges();
}
i need copy and paste some records of my table with just but change one field.
here is my code:
using (ClearWhiteDBEntities cwContext = new ClearWhiteDBEntities())
{
var qlstfld = from lstflds in cwContext.tblListFields
where lstflds.listId == theLongSrc
select lstflds;
foreach (var item in qlstfld)
{
tblListField lstFldRow = new tblListField
{
name = item.name,
filterFieldId = item.filterFieldId,
listId = theLongDes, //this field must be change in paste
continueById = item.continueById,
destinationId = item.destinationId,
conditionId = item.conditionId,
userId = userId,
date = Convert.ToDateTime(DateTime.Now.ToShortDateString()),
time = DateTime.Now.TimeOfDay,
IP = trueIp
};
cwContext.AddTotblListFields(lstFldRow);
cwContext.SaveChanges();
}
}
but i get this error:
an error accured white starting a transaction on the provider connection. see the inner exception for details.
what is best solution to copy and paste records but change one field?
If you are using change tracking:
using (ClearWhiteDBEntities cwContext = new ClearWhiteDBEntities())
{
var qlstfld = from lstflds in cwContext.tblListFields
where lstflds.listId == theLongSrc
select lstflds;
foreach (var item in qlstfld)
{
cwContext.ObjectStateManager.ChangeObjectState(item, System.Data.EntityState.Added);
item.Id = 0;
item.listId = theLongDes; //this field must be change in paste
}
cwContext.SaveChanges();
}