Develop Reference

Get all SDF/COS objects from PDF

I am trying to get a list of all SDF/COS objects within a PDF document, using PDFNet 7.0.4 and netcoreapp3.1. Using a different PDF parser, I know that this document has 570 total COS objects within it, including 3 images.
Initially I used PDFDoc to load the document, and iterated through the pages just looking for Element objects of type e_image or e_inline_image, but this only yielded 2 out of 3 images. In a larger document it did even worse; 0 out of ~2600 images.
Now, I've stepped back and am trying to do a lower level search via SDFDoc. I can get a trailer object, and then iterate through it, recursing any e_dict or e_stream objects, and returning anything that looks like a real object (i.e., anything that actually has an object number and generation).
IEnumerable<Obj> Recurse(Obj root)
{
var idHash = new HashSet<PdfIdentifier>();
return Recurse(root, idHash);
static IEnumerable<Obj> Recurse(Obj obj, HashSet<PdfIdentifier> idHash)
{
var id = obj.ToPdfIdentifier();
if (!idHash.Contains(id))
{
if (id != nullIdentifier)
{
idHash.Add(id);
yield return obj;
}
if (obj.GetType().OneOf(Obj.ObjType.e_dict, Obj.ObjType.e_stream))
{
for (var iter = obj.GetDictIterator(); iter.HasNext(); iter.Next())
{
foreach (var child in Recurse(iter.Value(), idHash))
{
yield return child;
}
}
}
}
}
}
static PdfIdentifier nullIdentifier = new PdfIdentifier() { Generation = 0, ObjectNum = 0 };
ToPdfIdentifier is a simple extension method to get the object number and generation:
public static PdfIdentifier ToPdfIdentifier(this pdftron.SDF.Obj obj) => new PdfIdentifier { ObjectNum = obj.GetObjNum(), Generation = obj.GetGenNum() };
This runs OK, but only returns 45 objects, none of them the images I'm actually interested in.
How can I simply get all COS objects from a document?
edit
Here is the original PDFDoc code we tried to get all images:
private IEnumerable<(PdfIdentifier id, Element el)> GetImages(Stream stream)
{
var doc = new PDFDoc(stream);
var reader = new ElementReader();
for (var iter = doc.GetPageIterator(); iter.HasNext(); iter.Next())
{
reader.Begin(iter.Current());
var el = reader.Next();
while (el != null)
{
var type = el.GetType();
if (el.GetType().OneOf(Element.Type.e_image, Element.Type.e_inline_image))
{
var obj = el.GetXObject();
var id = el.GetXObject().ToPdfIdentifier();
yield return (id, el);
}
el = reader.Next();
}
reader.End();
}
}
This kind of worked in that it returned some images, but not all. For some sample documents it returned all, for some it returned a subset, and for some it returned none at all.
edit
Just for future reference, thanks to the answer below from Ryan, we ended up with a pair of nice clean extension methods:
public static IEnumerable<SDF.Obj> GetAllObj(this SDF.SDFDoc sdfDoc)
{
var xrefTableSize = sdfDoc.XRefSize();
for (int objNum = 0; objNum < xrefTableSize; objNum++)
{
var obj = sdfDoc.GetObj(objNum);
if (obj.IsFree())
{
continue;
}
else
{
yield return obj;
}
}
}
and
public static string Subtype(this SDF.Obj obj) => obj.FindObj("Subtype") switch
{
null => null,
var s when s.IsName() => s.GetName(),
var s when s.IsString() => s.GetAsPDFText(),
_ => throw new Exception("COS object has an invalid Subtype entry")
};
Now we can get images as simply as sdfDoc.GetAllObj().Where(o => o.IsStream() && o.Subtype() == "Image"); or even use Linq:
from o in sdfDoc.GetAllObj()
where o.IsStream() && o.Subtype() == "Image"
select new Image(o);

If you want to get the images that are actually used on a page of the PDF (in case there happen to be unused images in the PDF), then you would use this sample code. This code would have the added bonus of including inline images.
https://www.pdftron.com/documentation/samples/dotnetcore/cs/ImageExtractTest
Though the above can be slow, if the document has hundreds or thousands of pages, that are complicated graphically.
The otherway, as you described, is to iterate the COS objects. The following C# code finds all Image streams. Note, the PDF standard specifically states that Streams have to be Indirect objects. So I think you can safely omit reading through all the direct objects.
using (PDFDoc doc = new PDFDoc("2002.04610.pdf"))
{
doc.InitSecurityHandler();
int xrefSz = doc.GetSDFDoc().XRefSize();
for (int xrefCounter = 0; xrefCounter < xrefSz; ++xrefCounter)
{
Obj o = doc.GetSDFDoc().GetObj(xrefCounter);
if (o.IsFree())
{
continue;
}
if(o.IsStream())
{
Obj subtypeObj = o.FindObj("Subtype");
if (subtypeObj != null)
{
string subtype = "";
if(subtypeObj.IsName()) subtype = subtypeObj.GetName();
if(subtypeObj.IsString()) subtype = subtypeObj.GetAsPDFText(); // Subtype should be a Name, but just in case
if (subtype.CompareTo("Image") == 0)
{
Console.WriteLine("Indirect object {0} is an Image Stream", o.GetObjNum());
}
}
}
}
}

Use LINQ expression to Update Property of object

So I have a situation where I need to dynamically update properties of my object based on values contained. In the case below, I need to update the value with replacing the first two characters of the current value with a different string if condition is true.
PersonDetail.EvaluateConditionalRule("ID",
"((ID.Length > Convert.ToInt32(#0) ) AND ID.Substring(Convert.ToInt32(#1), Convert.ToInt32(#2)) == #3 )",
new[] { "1", "0", "2", "SS" }, " ID = (#0 + ID.Substring(Convert.ToInt32(#1))) " , new[] { "98", "2" });
public static void EvaluateConditionalRule(this PersonDetail Detail, String PropertyToEvaluate,
String ConditionalExpression, String[] parameters, String IfTrueExpression,String[] IfTrueExpreassionparameters )
{
var property = Detail.GetType().GetProperties().Where(x => x.Name == PropertyToEvaluate).FirstOrDefault();
if (property == null)
throw new InvalidDataException(String.Format("Please specify a valid {0} property name for the evaluation.", Detail.GetType()));
//put together the condition like so
if (new[] { Detail }.AsQueryable().Where(ConditionalExpression, parameters).Count() > 0 && IfTrueExpression != null)
{
var result = new[] { Detail }.AsQueryable().Select(IfTrueExpression, IfTrueExpreassionparameters);
//Stuck Here as result does not contain expected value
property.SetValue( Detail,result , null);
}
}
Essentially what I want is, to be able to execute expressions fed this, and I don't think I have the format right for the substring replace expression to correctly evaluate. What I want from the above is something like
ID = "98"+ ID.Substring(2);
Any help will be appreciated. Thanks

Please give us some information on why you need the updates to be this dynamic. Do you want the user to enter the condition strings by hand?
About your code:
Your selector string is wrong. In the LINQ Dynamic Query Library there is a special syntax for it. Please look that up in the Documentation (see paragraph Expression Language and Data Object Initializers) provided with the sample: http://msdn.microsoft.com/en-US/vstudio/bb894665.aspx
I wrote a small sample:
var id = new string[] { "SS41231" }.AsQueryable();
// *it* represents the current element
var res = id.Where("it.Length > #0 AND it.Substring(#1, #2) = #3", 1, 0, 2, "SS"); // Save the result, don't throw it away.
if (res.Any())
{
// Line below in normal LINQ: string newID = res.Select(x => "98" + x.Substring(2)).First();
string newId = res.Select("#0 + it.Substring(#1)", "98", 2).Cast<string>().First();
Console.WriteLine(newId);
}
Please write some feedback.
Greetings.

Not sure what exactly you mean saying "dynamically" but i suppose the approach with lambdas does not fit you:
static void Main(string[] args) {
User user = new User { ID = 5, Name = "Test" };
SetNewValue(user, u => u.Name, s => s.StartsWith("T"), s => s + "123");
}
static void SetNewValue<TObject, TProperty>(TObject obj, Func<TObject, TProperty> propertyGetter, Func<TProperty, bool> condition, Func<TProperty, TProperty> modifier) {
TProperty property = propertyGetter(obj);
if (condition(property)) {
TProperty newValue = modifier(property);
//set via reflection
}
}
So I would recommend you using Expression trees which allow you to build any runtime construction you like, for part of your example
var exp = Expression.Call(Expression.Constant("test"), typeof(string).GetMethod("Substring", new[] { typeof(int) }), Expression.Constant(2));
Console.WriteLine(Expression.Lambda(exp).Compile().DynamicInvoke()); //prints "st"
However if you want to use strings with raw c# code as expressions check the CSharpCodeProvider class

Use a numeric value in a linq dynamic query string

I am trying to make a dynamic linq query that will check for values based on a string.
First of all, here's the query:
objQry = from o in m_Db.OBJECTS.Where(whereConditions)
select o;
if(!objQry.Any())
{
return null;
}
The whereConditions variable is a string I build and pass as parameter to find out the values I need. Here's examples of valid string:
OBJ_NAME == \"Sword\" and OBJ_OWNER == \"Stan\"
This will return any item whose name is "Sword" and owner is "Stan;
OBJ_COLOR == \"Blue\" OR OBJ_COLOR == \"Red\"
This will return any item which color is either blue or red.
Up to there, I'm fine, but now I have a problem: I need to check a decimal field. So I've tried this string:
OBJ_NUMBER == 1
But the query returns null even if there are objects which OBJ_NUMBER value is 1. It's a decimal. How can I indicate the query that they need to check for a decimal value?
**** EDIT ****
I have tried to "modify" the value passed so that it looks like this:
"CARD_NUMBER == Convert.ToDecimal(1)"
And now I have a different kind of error telling me this:
LINQ to Entities does not recognize the method 'System.Decimal ToDecimal(Int32)' method, and this method cannot be translated into a store expression.
Any clues anyone? I'm still looking for a way to do this. Thanks!
EDIT 2
You can get an example of how my code is shaped by looking at this question.
Let's come back at this problem. I want to check decimal values. Let's say that OBJ_NUMBER is a decimal field.
Using Dynamic Linq, I tried to read the decimal field. Say that I want to get each object which number is 1.27. The whereConditions field would then be shaped like this:
OBJ_NUMBER == 1.27
But then I would get an Invalid real literal '1.27' error. I don't know why.
So I have tried Gert Arnold's solution and done this instead:
decimal bDecimal = decimal.Parce(valueToParse);
param = new ObjectParameter("cardNumber", typeof(decimal)) { Value = bDecimal };
valuesToUse.Add("CARD_NUMBER == #cardNumber");
listParams.Add(param);
But I ended up having 2 problems:
The first problem is that my whereConditions string is shaped this way:
CARD_NUMBER == #cardNumber
But I get the following error:
No property or field 'cardNumber' exists in type 'CARD'
Leading me to believe that it cannot make the link between the object parameter and the string used to do the query.
As you can see, I have a list of Params. This is because I cannot know for sure how many parameters the user will chose. So each time the user enters a new search field, I have to create a new ObjectParameter and store it in a list. Here's how I try to do the thing after:
ObjectParameter[] arrayParameters = listParams.ToArray();
// Convert the list to an array
And then, when I try to make the query:
cardQry = from c in mDb.CARD.Where(whereConditions, arrayParameters)
select c;
But to no avail.
RESULTS
Based on the answered question below, I have developped something "awful", yet functional.
First of all, I ignore every decimal fields because I could never reach them with dynamic linq. Instead, I do this:
var valuesToParse = keyValuePair.Value.Split(new[] {' '}, StringSplitOptions.RemoveEmptyEntries);
// Here I parse the value and, if that's the case, the symbol.
decimal baseValue = decimal.Parse(valuesToParse[0]);
if (valuesToParse.Count() > 1)
{
string baseMethod = valuesToParse[1];
if (baseMethod == ">" || baseMethod == ">=")
{
if (baseMethod == ">=")
{
baseValue--;
}
// The list is actually like this: Dictionary<string, object> list = new Dictionary<string, object>();
list.Add("low", baseValue);
// I kind of activate a tag telling me that the user is looking for a higher value.
cardHigher = true;
}
else
{
if (baseMethod == "<=")
{
baseValue++;
}
list.Add("low", baseValue);
cardLower = true;
}
}
else
{
//lowParam = new ObjectParameter("dec", typeof(decimal)) { Value = baseValue };
list.Add("low", baseValue);
}
cardNumberActivated = true;
At the end, when I get the list of objects, I do this:
if (list.Count > 0)
{
(example)
if (cardNumberActivated)
{
if (cardHigher)
{
q = mDb.CARD.Where("CARD_NUMBER >= #0", list["low"]).ToList();
}
else if (cardLower)
{
q = mDb.CARD.Where("CARD_NUMBER <= #0", list["low"]).ToList();
}
else
{
q = mDb.CARD.Where("CARD_NUMBER == #0", list["low"]).ToList();
}
}
}
// Here we get the orinalData with the basic filters.
listToReturn.AddRange(cardQry);
if (q != null)
{
//listToReturn.AddRange(q);
for (int i = 0; i < listToReturn.Count; i++)
{
var priceList1 = listToReturn[i];
if (!q.Any(_item => _item.CARD_NUMBER == priceList1.CARD_NUMBER))
{
listToReturn.RemoveAt(i);
i--;
}
}
}
And it works. This is not an elegant way to make it work, but I can validate the fields the way I wanted, and for this, I am thankful at last.

You should not build a query string with inline predicate values. Use parameters in stead. Then will also be able to specify the type:
var whereConditions= "it.CARD_NUMBER = #cardNumber";
var param = new ObjectParameter("cardNumber", typeof(decimal)) { Value = 1 };
objQry = from o in m_Db.OBJECTS.Where(whereConditions, param);
Edit
I don't know what doesn't work in your code. Here's just a random piece of working code derived from one of my own projects:
var param1 = new ObjectParameter("dec", typeof(decimal)) { Value = 90000m };
var param2 = new ObjectParameter("int", typeof(int)) { Value = 90000 };
var q = ValueHolders.Where("it.DecimalValue >= #dec OR it.IntegerValue > #int",
param1, param2).ToList();
Note that param1, param2 could also be an array of ObjectParameter.

Why failed to add item on a session IEnumerable list

Work on entity framework vs2010,
I want to store somewhere some set of objects obtained from the database. Because I dont want to call DB after which user request. And I do it this way:
public IEnumerable<Message> Messages {
get { return HttpContext.Session[SESSION_MESSAGES_NAME] as IEnumerable<Message>; }
set { HttpContext.Session[SESSION_MESSAGES_NAME] = value; }
}
objEntity = new CmnItemSpecificationDetail();
objEntity.ItemSpecificationDetailRecordID = hidItemSpecificationDetailRecordID.Value.ToString() == "" ? _ItemSpecificationDetail.Count + 1 : int.Parse(hidItemSpecificationDetailRecordID.Value.ToString());
objEntity.SpecificationID = drpSpecification.SelectedIndex == 0 ? -1 : int.Parse(drpSpecification.SelectedValue);
objEntity.SpecValue = Convert.ToDecimal(txtSpecValue.Text);
objEntity.UOMID = drpUOM.SelectedIndex == 0 ? -1 : int.Parse(drpUOM.SelectedValue);
objEntity.UOMID = 1;
objEntity.Action = Entity.ActionMode.Add;
objEntity.Action = Entity.ActionMode.Add;
objEntity.CreateBy = LogInUser;
objEntity.CreateOn = DateTime.Now;
objEntity.CreatePc = PCName;
Messages.Append(objEntity);//failed to add item
public static class exten
{
public static IEnumerable<T> Append<T>(this IEnumerable<T> source, params T[] items)
{
return source.Concat(items);
}
}
This list failed to fill up item,want to know why this can not add item on list,what is the problem,How to solve this problem.
If have any query please ask ,thanks in advanced.

You may want to modify your Getter a bit.
The way it currently works it doesn't handle situations where the Messages is null (hasn't been initialized).
Change this
get { return HttpContext.Session[SESSION_MESSAGES_NAME] as IEnumerable<Message>; }
To this.
get { var msg = HttpContext.Session[SESSION_MESSAGES_NAME] as IEnumerable<Message>;
if(msg == null) {
msg = new List<Message>();
HttpContext.Session[SESSION_MESSAGES_NAME] = msg;
}
return msg;
}

Your Append() extension method does NOT change the original sequence. It returns a new sequence with the specified items appended to it.
When you call Messages.Append(objEntity), you don't do anything with the return value, so you are throwing away the results of the Append().
To fix it, you will need to do something with the return value. Perhaps just do:
Messages = Messages.Append(objEntity);

Lucene.NET and searching on multiple fields with specific values

I've created an index with various bits of data for each document I've added, each document can differ in it field name.
Later on, when I come to search the index I need to query it with exact field/ values - for example:
FieldName1 = X AND FieldName2 = Y AND FieldName3 = Z
What's the best way of constructing the following using Lucene .NET:
What analyser is best to use for this exact match type?
Upon retrieving a match, I only need one specific field to be returned (which I add to each document) - should this be the only one stored?
Later on I'll need to support keyword searching (so a field can have a list of values and I'll need to do a partial match).
The fields and values come from a Dictionary<string, string>. It's not user input, it's constructed from code.
Thanks,
Kieron

Well, I figured it out eventually - here's my take on it (this could be completely wrong, but it works for):
public Guid? Find (Dictionary<string, string> searchTerms)
{
if (searchTerms == null)
throw new ArgumentNullException ("searchTerms");
try
{
var directory = FSDirectory.Open (new DirectoryInfo (IndexRoot));
if (!IndexReader.IndexExists (directory))
return null;
var mainQuery = new BooleanQuery ();
foreach (var pair in searchTerms)
{
var parser = new QueryParser (
Lucene.Net.Util.Version.LUCENE_CURRENT, pair.Key, GetAnalyzer ());
var query = parser.Parse (pair.Value);
mainQuery.Add (query, BooleanClause.Occur.MUST);
}
var searcher = new IndexSearcher (directory, true);
try
{
var results = searcher.Search (mainQuery, (Filter)null, 10);
if (results.totalHits != 1)
return null;
return Guid.Parse (searcher.Doc (results.scoreDocs[0].doc).Get (ContentIdKey));
}
catch
{
throw;
}
finally
{
if (searcher != null)
searcher.Close ();
}
}
catch
{
throw;
}
}