I've used NEST for elasticsearch for a while now and up until now I've used the regular ElasticSearchClient.Index(...) function, but now I want to index many items in a bulk operation.
I found the IndexMany(...) function, but I must do something wrong because nothing is added to the elastic search database as it does with the regular Index(...) function?
Does anyone have any idea?
Thanks in advance!
I found the problem. I had to specifiy the index name in the call to IndexMany
var res = ElasticClient.CreateIndex("pages", i => i.Mappings(m => m.Map<ESPageViewModel>(mm => mm.AutoMap())));
var page = new ESPageViewModel
{
Id = dbPage.Id,
PageId = dbPage.PageId,
Name = dbPage.Name,
Options = pageTags,
CustomerCategoryId = saveTagOptions.CustomerCategoryId,
Link = dbPage.Link,
Price = dbPage.Price
};
var pages = new List<ESPageViewModel>() { page };
var res2 = ElasticClient.IndexManyAsync<ESPageViewModel>(pages, "pages");
This works as expected. Guess I could specify a default index name in the configuration to avoid specifying the index for the IndexMany call.
If you are using C# you should create a list of objects that you want to insert then call the IndexMany function.
Example :
List<Business> businessList = new List<Business>();
#region Fill the business list
...............................
#endregion
if (businessList.Count == 1000) // the size of the bulk.
{
EsClient.IndexMany<Business>(businessList, IndexName);
businessList.Clear();
}
And in the end check again
if (businessList.Count > 0)
{
EsClient.IndexMany<Business>(businessList, IndexName);
}
Related
I have to take the same values from multiple source and so I used Concat but I have large number of fields and couple of more sources too.
IEnumerable<Parts> partsList = (from parts in xml.XPathSelectElements("//APS/P")
select new WindchillPart
{
Code = (string)parts.Element("Number"),
Part = (string)parts.Element("KYZ"),
Name = (string)parts.Element("Name"),
})
.Concat(from uparts in xml.XPathSelectElements("//APS/U")
select new WindchillPart
{
Code = (string)uparts.Element("Number"),
Part = (string)uparts.Element("KYZ"),
Name = (string)uparts.Element("Name"),
});
I almost have 15 fields and 5 sources. So is there anyway to make the fields as common and just add the sources somewhere to work and simplify this?
You could create an array of all your pathes, and use SelectMany to get the elements. In the end, you call Select just once:
string[] pathes = new string[] { "//APS/P", "//APS/U" };
IEnumerable<Parts> partsList = pathes.SelectMany(path => xml.XPathSelectElements(path)).
Select(uparts => new WindchillPart
{
Code = (string)uparts.Element("Number"),
Part = (string)uparts.Element("KYZ"),
Name = (string)uparts.Element("Name"),
});
I have a simple class which holds a primary key of which I don't know what type it will be before it runs, as i'm getting the data from COM. It will either be an int or string.
I basically just need to fill up my toUpdateList & toAddList. This was working fine below with not too many records to play around with. However now the mongoDBList returns around 65k records and it's all turned very slow and it's taking 15+ minutes to resolve toUpdateList.
I'm pretty new to C# so I'm likely missing something.
I basically just need to compare one list to another and see if the RecordRevision is higher in the toUpdateList. For the toAddList this ones pretty simple as if it doesn't exist it needs to be added.
Thanks for looking I appreciate it!
class KeyRevision
{
public dynamic RecordRevision;
public dynamic PrimaryKey;
}
List<KeyRevision> potentialUpdateList = new List<KeyRevision>();
List<KeyRevision> mongoDBList = new List<KeyRevision>();
List<KeyRevision> toUpdateList = new List<KeyRevision>();
List<KeyRevision> toAddList = new List<KeyRevision>();
var sql = env.ExecuteSQL(sqlQuery);
sql.First();
// Loop over them and add to array
do
{
if (sql.RecordCount > 0)
{
//Console.WriteLine(sql.GetPropertyValue(primaryKey).ToString() + " : " + sql.RecordRevision);
var o = new KeyRevision();
o.PrimaryKey = sql.GetPropertyValue(primaryKey);
o.RecordRevision = sql.RecordRevision;
potentialUpdateList.Add(o);
}
sql.Next();
} while (!sql.EOF);
// Ask mongo for docs
var docs = collection1.Find(_ => true).Project("{RecordRevision: 1}").ToList();
// Get them into our type
mongoDBList = docs.ConvertAll(x => new KeyRevision()
{
PrimaryKey = x.GetValue("_id"),
RecordRevision = x.GetValue("RecordRevision")
});
// Finds which records we need to update
toUpdateList = potentialUpdateList.Where(x =>
mongoDBList.Any(y => y.PrimaryKey == x.PrimaryKey && y.RecordRevision < x.RecordRevision)).ToList();
// Finds the records we need to add
toAddList = potentialUpdateList.Where(x =>
mongoDBList.FindIndex(y => y.PrimaryKey == x.PrimaryKey) < 0).ToList();
Console.WriteLine($"{toUpdateList.Count} need to be updated");
Console.WriteLine($"{toAddList.Count} need to be updated");
query is like this:-
labRequestItem.LabTest.Select(p=>p.LabTestICDCodes= labRequests.ICDCodes.Select(t => new ICDCodeList()
{
ICDCodeId = t.ICDCodeId,
ICDCode = t.ICDCodeName,
ICDDescription = t.ICDDescription,
ICDCodeType = t.ICDCodeType
}).ToArray());
Now the issue is that LabTestICDCodes is not updating with the new list value returning from select statement
I have tested in the Linqpad, following shall work, but please note:
finalResult would be List of type of LabTestICDCodes (which is again IEnumerable itself) with new / modified values, as that's the only column projected. You have to project the whole object you want as a result, values will modify as expected
var finalResult =
labRequestItem.LabTest.Select(p=>p.LabTestICDCodes= labRequests.ICDCodes.Select(t => new ICDCodeList()
{
ICDCodeId = t.ICDCodeId,
ICDCode = t.ICDCodeName,
ICDDescription = t.ICDDescription,
ICDCodeType = t.ICDCodeType
}).ToArray()).ToList();
I found only way to solve this problem by doing following step:-
1.Get the value in local variable
var icdList = labRequests.ICDCodes.Select(t => new ICDCodeList()
{
ICDCodeId = t.ICDCodeId,
ICDCode = t.ICDCodeName,
ICDDescription = t.ICDDescription,
ICDCodeType = t.ICDCodeType
}).ToArray();
Now assign the value by usinf foreach loop.
foreach (var test in labRequestItem.LabTest)
{
test.LabTestICDCodes = icdList;
}
In Last step I have to use foreach loop, if any other solution then
please reply otherwise I will use the same as it mentioned above.
I'm using C#, EF5, and Lambda style queries against SQL.
I have the usual scenario of binding data to gridviews. Some of the results for my columns may be too long (character count) and so I only want to display the first 'n' characters. Let's say 10 characters for this example. When I truncate a result, I'd like to indicate this by appending "...". So, let's say the following last names are returned:
Mercer, Smith, Garcia-Jones
I'd like them to be returned like this:
Mercer, Smith, Garcia-Jon...
I was doing something like this:
using (var context = new iaiEntityConnection())
{
var query = context.applications.Where(c => c.id == applicationPrimaryKey);
var results = query.ToList();
foreach (var row in results)
{
if (row.employerName.Length > 10)
{
row.employerName = row.employerName.Substring(0, Math.Min(10, row.employerName.ToString().Length)) + "...";
}
if (row.jobTitle.Length > 10)
{
row.jobTitle = row.jobTitle.Substring(0, Math.Min(10, row.jobTitle.ToString().Length)) + "...";
}
}
gdvWorkHistory.DataSource = results;
gdvWorkHistory.DataBind();
However, if I change my query to select specific columns like this:
var query2 = context.applications.Select(c => new
{
c.id,
c.applicationCode,
c.applicationCategoryLong,
c.applicationType,
c.renew_certification.PGI_nameLast,
c.renew_certification.PGI_nameFirst,
c.renew_certification.PAI_homeCity,
c.renew_certification.PAI_homeState,
c.reviewStatusUser,
c.dateTimeSubmittedByUser
})
The result appears to become read-only if specific columns are selected, and I really should be selecting just the columns I need. I'm losing my ability to edit the result set.
So, I'm rethinking the entire approach. There must be away to select the first 'n' characters on select, right? Is there anyway to append the "..." if the length is > 10 on select? That seems trickier. Also, I guess I could parse through the gridview after bind and make this adjustment. Or, perhaps there is a way to maintain my ability to edit the result set when selecting specific columns?
I welcome your thoughts. Thanks!
To quote MSDN
Anonymous types provide a convenient way to encapsulate a set of read-only properties into a single object without having to explicitly define a type first.
So you would have to define a class and select into that if you want read write capability.
e.g.
public class MyClass {
public int id { get; set; }
public string applicationCode {get; set; }
// rest of property defintions.
}
var query2 = context.applications.Select(c => new MyClass {
id = c.id,
applicationCode = c.applicationCode,
// Rest of assignments
};
As to just providing 10 character limit with ... appended. I'm going to assume you mean on the applicationcategoryLog field but you can use the same logic on other fields.
var query2 = context.applications.Select(c => new
{
c.id,
c.applicationCode,
applicationCategoryLong = (c.applicationCategoryLong ?? string.Empty).Length <= 10 ?
c.applicationCategoryLong :
c.applicationCategoryLong.Substring(0,10) + "...",
c.applicationType,
c.renew_certification.PGI_nameLast,
c.renew_certification.PGI_nameFirst,
c.renew_certification.PAI_homeCity,
c.renew_certification.PAI_homeState,
c.reviewStatusUser,
c.dateTimeSubmittedByUser
})
I have method that looks like this:
private static IEnumerable<OrganizationViewModel> GetOrganizations()
{
var db = new GroveDbContext();
var results = db.Organizations.Select(org => new OrganizationViewModel
{
Id = org.OrgID,
Name = org.OrgName,
SiteCount = org.Sites.Count(),
DbSecureFileCount = 0,
DbFileCount = 0
});
return results;
}
This is returns results pretty promptly.
However, you'll notice the OrganizationViewModel has to properties which are getting set with "0". There are properties in the Organization model which I added via a partial class and decorated with [NotMapped]: UnsecureFileCount and SecureFileCount.
If I change those 0s to something useful...
DbSecureFileCount = org.SecureFileCount,
DbFileCount = org.UnsecureFileCount
... I get the "Only initializers, entity members, and entity navigation properties are supported" exception. I find this a little confusing because I don't feel I'm asking the database about them, I'm only setting properties of the view model.
However, since EF isn't listening to my argument I tried a different approach:
private static IEnumerable<OrganizationViewModel> GetOrganizations()
{
var db = new GroveDbContext();
var results = new List<OrganizationViewModel>();
foreach (var org in db.Organizations)
{
results.Add(new OrganizationViewModel
{
Id = org.OrgID,
Name = org.OrgName,
DbSecureFileCount = org.SecureFileCount,
DbFileCount = org.UnsecureFileCount,
SiteCount = org.Sites.Count()
});
}
return results;
}
Technically this gives me the correct results without an exception but it takes forever. (By "forever" I mean more than 60 seconds whereas the first version delivers results in under a second.)
Is there a way to optimize the second approach? Or is there a way to get the first approach to work?
Another option would be to load the values back as an anonymous type and the loop through those to load your viewmodel (n+1 is most likely the reason for the slowness).
For example:
var results = db.Organizations.Select(org => new
{
Id = org.OrgID,
Name = org.OrgName,
DbSecureFileCount = org.SecureFileCount,
DbFileCount = org.UnsecureFileCount,
SiteCount = org.Sites.Count()
}).ToList();
var viewmodels = results.Select( x=> new OrganizationViewModel
{
Id = x.Id,
Name = x.Name,
DbSecureFileCount = x.DbSecureFileCount,
DbFileCount = x.DbFileCount,
SiteCount = x.SiteCount
});
Sorry about the formatting; I'm typing on a phone.
You are basically lazy loading each object at each iteration of the loop, causing n+1 queries.
What you should do is bring in the entire collection into memory, and use it from there.
Sample code:
var organizationList = db.Organizations.Load();
foreach (var org in organizationList.Local)
{
//Here you are free to do whatever you want
}