I'm trying to fetch data from the database and export the data to a csv, I can achieve that, but my concern is having code that goes to the database within a for each loop, to get the data that has a relationship with the record I have already. What is the best way to implement this to avoid trips to the database within a foreach loop?
public async Task<FileResult> FetchAllStudents(int SchoolId)
{
var studentList = await student.GetAll(SchoolId);
if (studentList != null)
{
StringBuilder sb = new();
//Append new line character.
sb.Append("\r\n");
foreach (var item in studentList )
{
//Get address of each student in the list
var studentAddress = await address.GetAddressByStudentId(item.StudentId);
var addressItem = studentAddress.Where(m => m.IsPrimary = true).FirstOrDefault();
var data = $"{addressItem.AddressLine1},{item.StudentName}";
//Append data with comma(,) separator.
sb.Append(data);
}
return File(Encoding.ASCII.GetBytes(sb.ToString()), "text/csv", "file.csv");
}
return null;
}
Sound like you want to fetch data in once and you get all the data without to query muiltple time to database. It is great to implement performance of an application. My idea as follow:
Get list of id of student.
Define a function inside the address services to such as "GetListOfStudentByIds" fetching data by list id of student. You can find out Any in Linq to working with array.
You dont need this line,
**
var addressItem = studentAddress.Where(m => m.IsPrimary = true).FirstOrDefault();
**
it makes application slower. You can add conditional while fetch data from GetListOfStudentByIds.
Thanks
Related
I have more than 15000 POCO elements stored in a Redis List. I'm using ServiceStack in order to save and get them. However, I'm not pleased about the response times that I have when I get them into a grid. As I read , it would be better to store these object in hash - but unfortunately I could not find any good example for my case :(
This is the method I use, in order to get them into my grid
public IEnumerable<BookingRequestGridViewModel> GetAll()
{
try
{
var redisManager = new RedisManagerPool(Global.RedisConnector);
using (var redis = redisManager.GetClient())
{
var redisEntities = redis.As<BookingRequestModel>();
var result =redisEntities.Lists["BookingRequests"].GetAll().Select(z=> new BookingRequestGridViewModel
{
CreatedDate =z.CreatedDate,
DropOffBranchName =z.DropOffBranch !=null ? z.DropOffBranch.Name : string.Empty,
DropOffDate =z.DropOffDate,
DropOffLocationName = z.DropOffLocation != null ? z.DropOffLocation.Name : string.Empty,
Id =z.Id.Value,
Number =z.Number,
PickupBranchName =z.PickUpBranch !=null ? z.PickUpBranch.Name :string.Empty,
PickUpDate =z.PickUpDate,
PickupLocationName = z.PickUpLocation != null ? z.PickUpLocation.Name : string.Empty
}).OrderBy(z=>z.Id);
return result;
}
}
catch (Exception ex)
{
return null;
}
}
Note that I use redisEntities.Lists["BookingRequests"].GetAll() which is causing performance issues (I would like to use just redisEntities.Lists["BookingRequests"] but I lose last updates from grid - after editing)
I would like to know if saving them into list is a good approach as for me it's very important to have a fast grid (I have now 1 second at paging which is huge).
Please, advice!
Firstly you should not create a new Redis Client Manager like RedisManagerPool instance each time, there should only be a singleton instance of RedisManagerPool in your App which all clients are resolved from.
But otherwise I would rethink your data access strategy, downloading 15K items in a batch is not an ideal strategy. You can create indexes by storing ids in Sets or you could store items in a sorted set with a value that you can page against like an incrementing id, e.g:
var redisEntities = redis.As<BookingRequestModel>();
var bookings = redisEntities.SortedSets["bookings"];
foreach (var item in new BookingRequestModel[0])
{
redisEntities.AddItemToSortedSet(bookings, item, item.Id);
}
That way you will be able to fetch them in batches, e.g:
var batch = bookings.GetRangeByLowestScore(fromId, toId, skip, take);
I have a DynamoDb table (named Fruit) with the following properties:
FruitId - string
CreatedDate - date
Type - number
Payload - blob
I also have a local list of strings List<string> fruitIds;.
I want to query the Fruit table and get only the Ids that have a corresponding record (i.e. exist) in the table.
What is a good way of doing that? Right now, I am looping over each Id in fruitIds and making a separate query to DyanmoDb to see if I get a record back, if I do, I then save that Id to another local variable called fruitIdsThatExistInDyanmoDb.
Is there a better way?
public IQueryable <fruits> GetAllfruitsIDs() {
return fruits.AsQueryable();
}
var data = GetAllfruitsIDs();
// Or u can use this :
public IEnumerable<fruits> GetAllfruitsIDs() {
return fruits.AsQueryable().ToList;
}
var data = GetAllfruitsIDs();
Using Linq, its very simple, just check if item's FruitId is in fruitIds:
var result = fruits.Where(f => fruitIds.Contains(f.FruitId));
to save their Ids in a new local variable as you said:
List<string> fruitIdsThatExistInDyanmoDb = fruits.Where(f => fruitIds.Contains(f.FruitId))
.Select(f=> f.FruitId).ToList();
I have this method that saves an entity with its related items (many-to-many relationship),
private static void Save<T>(TbCommonHistoryLog log, List<T> lstDetails) where T : IHasSerial
{
foreach (var item in lstDetails.OrderBy(x => x.Serial))
{
var ser = SerializeObject(item);
var record = oContext.TbHistoryLog_Lists.FirstOrDefault(x => x.ListObjectJson == ser);
if (record == null) //add new list item
{
TbCommonHistoryLog_Lists listObject = new TbCommonHistoryLog_Lists()
{
ListObjectJson = SerializeObject(item)
};
var details = new TbCommonHistoryLogDetails { TbHistoryLog = log, TbHistoryLog_Lists = listObject };
oContext.TbHistoryLogDetails.Add(details);
}
else //attach an existing list item
{
var o = oContext.TbHistoryLog_Lists.Find(record.Id);
oContext.TbHistoryLog_Lists.Attach(o);
var details = new TbCommonHistoryLogDetails { TbHistoryLog = log, TbHistoryLog_Lists = o };
oContext.TbHistoryLogDetails.Add(details);
}
}
oContext.BulkSaveChanges();
}
I have two tables: TbCommonHistoryLog, TbCommonHistoryLog_Lists, that are in many to many relationship, the joining table is TbCommonHistoryLogDetails,
What I'm doing here is an auditing for master-detail models, all audits are serialized to JSON in DB, I save the head object in the TbCommonHistoryLog table, and every list item in the TbHistoryLog_Lists table, in the mthod above I check if the list item is already exists in the database or not to avoid duplicating.
but this process takes more than 15 seconds which is a very long time, I can't figure out what am I doing wrong here.. please help?
For every single item in collection you're querying database. My suggestion is to save records in var, then ask the variable if the item is in database.
var databaseRecords = oContext.TbHistoryLog_Lists.ToList();
Then in the loop:
var record = databaseRecords.FirstOrDefault(x => x.ListObjectJson == ser);
I am trying to delete documents ids from a collection in RavenDB using the below logic
var documentstore = new DocumentStore
{
Url = "http://localhost:8080",
DefaultDatabase = "Employee"
};
documentstore.Initialize();
using (var session = documentstore.OpenSession())
{
var test = new List<string>();
test.Add("emps/81993");
test.Add("emps/40319");
var w1 = session.Load<Employee>(test);
session.Delete(w1);
session.SaveChanges();
}
I get the below error
Models.Employee[] is not associated with the session, cannot delete unknown
How do I go about doing a bulk delete of document Ids from the collection?
Thanks
You are trying to delete the array of employees, not each employee itself. When you pass in an Enumerable in the Load, you will get an array back containing each entity.
Try this instead:
using (var session = documentstore.OpenSession())
{
var test = new List<string>();
test.Add("emps/81993");
test.Add("emps/40319");
Employee[] employees = session.Load<Employee>(test);
foreach (var employee in employees)
{
session.Delete(employee);
}
session.SaveChanges();
}
To further explain, the returned array is not tracked by RavenDb's UoW, but each individual item in the array is, that's why you get the message about Employee[] is not associated with the session.
You can use set based operation.
The following code will query specific index and delete everything matching the query
var operation = store.DatabaseCommands.DeleteByIndex("Index Name", new IndexQuery
{
Query = "Lucene query of stuff to delete"
}, options: null);
for example, the following code will delete all documents from a document collection called "Companies" (excerpt from one of RavenDB unit tests)
var operation = documentStore.DatabaseCommands.DeleteByIndex("Raven/DocumentsByEntityName", new IndexQuery
{
Query = "Tag:[[Companies]]"
});
You can read more in this link to RavenDB documentation
This section simply reads from an excel spreadsheet. This part works fine with no performance issues.
IEnumerable<ImportViewModel> so=data.Select(row=>new ImportViewModel{
PersonId=(row.Field<string>("person_id")),
ValidationResult = ""
}).ToList();
Before I pass to a View I want to set ValidationResult so I have this piece of code. If I comment this out the model is passed to the view quickly. When I use the foreach it will take over a minute. If I hardcode a value for item.PersonId then it runs quickly. I know I'm doing something wrong, just not sure where to start and what the best practice is that I should be following.
foreach (var item in so)
{
if (db.Entity.Any(w => w.ID == item.PersonId))
{
item.ValidationResult = "Successful";
}
else
{
item.ValidationResult = "Error: ";
}
}
return View(so.ToList());
You are now performing a database call per item in your list. This is really hard on your database and thus your performance. Try to itterate trough your excel result, gather all users and select them in one query. Make a list from this query result (else the query call is performed every time you access the list). Then perform a match between the result list and your excel.
You need to do something like this :
var ids = so.Select(i=>i.PersonId).Distinct().ToList();
// Hitting Database just for this time to get all Users Ids
var usersIds = db.Entity.Where(u=>ids.Contains(u.ID)).Select(u=>u.ID).ToList();
foreach (var item in so)
{
if (usersIds.Contains(item.PersonId))
{
item.ValidationResult = "Successful";
}
else
{
item.ValidationResult = "Error: ";
}
}
return View(so.ToList());