In my .NET application I have an XmlWriterTraceListener, is there any thing in .NET that already aids in reading these files as enumerated models?
I haven't found anything yet and am thinking I'll just need to open the file as read only with a object shared for read and writing so I don't lock the file for further logging then use linq to XML or an XML Reader to read the elements.
I think you will have to get your hands dirty with LINQ to XML or with Microsoft's LogParser. It's a free tool and it can be integrated with your applications. It has a nice SQL-like language for extracting information.
Related
I am looking for an LDIF parser for C#. I am trying to parse an LDIF file so that I can check objects don't exist before adding them. Adding them when the already exist using ntdsSchemaAdd) causes an entry in the error logs.
A quick websearch revealed: http://wiki.github.com/skradel/Zetetic.Ldap/. They have provided a .net API.
From the page:
Zetetic.Ldap is a .NET library for
.NET 2 and above, which makes it
easier to work with directory servers
(like Active Directory, ADAM, Red Hat
Directory Server, and others). Some of
the key features of Zetetic.Ldap are:
1.LDIF file parsing and generation – Read and write the file format used
for moving data around between
directory systems
2.LDAP Entry-oriented API with change tracking – Create and modify directory
objects in a more natural way
3.LDAP Schema interrogation – Quick programmatic access to the kinds of
objects and fields your directory
server understands. Learn if an
attribute is a string, a number, a
date, etc., without lots of manual
research and re-parsing
4.LDIF Pivoter – Turn an LDIF file into a (comma or tab-delimited) flat
file for analysis or loading into
systems that don’t speak LDIF We built
the Zetetic.Ldap library to make
directory projects and programming
faster and easier, and release it here
in the hopes that others will find it
useful too. As far as we know, this is
the only .NET library that really
understands the LDIF specification.
Download link: http://github.com/downloads/skradel/Zetetic.Ldap/Zetetic.Ldap_20090831.zip
I would parse it myself.
If you look at the LDIF RFC for the EBNF, you'll see that it's not a very complex grammar.
I've parsed a large amount of LDIF before using Regexes reliably. Though your mileage may vary.
I want to pass an XML object from code behind file of an aspx to an class library.for that how can i create a XML Object.
please its urgent.
.NET includes multiple XML APIs (XML Document—a typical DOM implementation, a streaming API, an XPath orientated API and LINQ to XML). So lots to chose from.
Without more detail impossible to say which is your best approach. I would suggest starting reading MSDN at "XML Documents and Data".
Load an XML file from disk
http://msdn.microsoft.com/en-us/library/875kz807.aspx
or some XML from a string
http://msdn.microsoft.com/en-us/library/system.xml.xmldocument.loadxml.aspx
I need to do some processing on fairly large XML files ( large here being potentially upwards of a gigabyte ) in C# including performing some complex xpath queries. The problem I have is that the standard way I would normally do this through the System.XML libraries likes to load the whole file into memory before it does anything with it, which can cause memory problems with files of this size.
I don't need to be updating the files at all just reading them and querying the data contained in them. Some of the XPath queries are quite involved and go across several levels of parent-child type relationship - I'm not sure whether this will affect the ability to use a stream reader rather than loading the data into memory as a block.
One way I can see of making it work is to perform the simple analysis using a stream-based approach and perhaps wrapping the XPath statements into XSLT transformations that I could run across the files afterward, although it seems a little convoluted.
Alternately I know that there are some elements that the XPath queries will not run across, so I guess I could break the document up into a series of smaller fragments based on it's original tree structure, which could perhaps be small enough to process in memory without causing too much havoc.
I've tried to explain my objective here so if I'm barking up totally the wrong tree in terms of general approach I'm sure you folks can set me right...
XPathReader is the answer. It isn't part of the C# runtime, but it is available for download from Microsoft. Here is an MSDN article.
If you construct an XPathReader with an XmlTextReader you get the efficiency of a streaming read with the convenience of XPath expressions.
I haven't used it on gigabyte sized files, but I have used it on files that are tens of megabytes, which is usually enough to slow down DOM based solutions.
Quoting from the below: "The XPathReader provides the ability to perform XPath over XML documents in a streaming manner".
Download from Microsoft
Gigabyte XML files! I don't envy you this task.
Is there any way that the files could be sent in a better way? E.g. Are they being sent over the net to you - if they are then a more efficient format might be better for all concerned. Reading the file into a database isn't a bad idea but it could be very time consuming indeed.
I wouldn't try and do it all in memory by reading the entire file - unless you have a 64bit OS and lots of memory. What if the file becomes 2, 3, 4GB?
One other approach could be to read in the XML file and use SAX to parse the file and write out smaller XML files according to some logical split. You could then process these with XPath. I've used XPath on 20-30MB files and it is very quick. I was originally going to use SAX but thought I would give XPath a go and was surprised how quick it was. I saved a lot of development time and probably only lost 250ms per query. I was using Java for my parsing but I suspect there would be little difference in .NET.
I did read that XML::Twig (A Perl CPAN module) was written explicitly to handle SAX based XPath parsing. Can you use a different language?
This might also help https://web.archive.org/web/1/http://articles.techrepublic%2ecom%2ecom/5100-10878_11-1044772.html
http://msdn.microsoft.com/en-us/library/bb387013.aspx has a relevant example leveraging XStreamingElement.
In order to perform XPath queries with the standard .NET classes the whole document tree needs to be loaded in memory which might not be a good idea if it can take up to a gigabyte. IMHO the XmlReader is a nice class for handling such tasks.
It seems that you already tried using XPathDocument and could not accomodate the parsed xml document in memory.
If this is the case, before starting to split the file (which is ultimately the right decision!) you may try using the Saxon XSLT/XQuery processor. It has a very efficient in-memory representation of a loaded XML document (the "tinytree" model). In addition Saxon SA (the shema-aware version, which isn't free) has some streaming extensions. Read more about this here.
How about just reading the whole thing into a database and then work with the temp database? That might be better because then your queries can be done more efficiently using TSQL.
I think the best solution is to make your own xml parser that can read small chunks not the whole file, or you can split the large file into small files and use dotnet classes with these files.
The problem is you can not parse some of data till the whole data is available so I recommend to use your own parser not dotnet classes
Have you been trying XPathDocument?
This class is optimized for handling XPath queries efficiently.
If you cannot handle your input documents efficiently using XPathDocument you might consider preprocessing and/or splitting up your input documents using an XmlReader.
You've outlined your choices already.
Either you need to abandon the XPath and use XmlTextReader or you need to break the document up into managable chunks on which you can use XPath.
If you choose the latter use XPathDocument its readonly restriction allows better used of memory.
Since in your case the data size can run in Gbs have you considered using ADO.NET with XML as a database. In addition to that the memory footprint would not be huge.
Another approach would be using Linq to XML with using elements like XElementStream. Hope this helps.
What are the best functions, practices, and/or techniques to read/write XML with C#/.NET?
If you are working with .NET 3.5, LINQ to XML is certainly a very good way to interact with XML.
MSDN Link
There are classes to read XML:
XmlDocument is slow and memory-intensive: it parses the XML and loads it into an in-RAM DOM, which is good if you want to edit it.
XmlReader is less memory-intensive: it scans the XML from front to back, never needing to keep all of it in RAM at once.
Similarly, for writing you can construct an XmlDocument and then save it, or use an XmlWriter.
After I wrote the above, there's now a new set of APIs which are easier to use: i.e. for example the XDocument and XElement classes.
By far the simplest method I've found for dealing with XML in C# is to use the XML Serialization tools. For example: http://www.dotnetjohn.com/articles.aspx?articleid=173.
Essentially, you can define C# classes that match your XML file (in fact, you can have them created for you if you have an XML definition file) and then you simply initialize instances of those classes directly from the XML file. Once you have them as instances, you can manipulate them as you wish and rewrite them back into XML files just as easily.
In a performance critical application XmlReader/XmlWriter are a good choice (see here) for the sake of simplicity which is offered by Linq to XML and XmlDocument.
I've found the MvpXml project very useful in past scenarios where performance is a consideration. There's a wealth of knowledge about good practice within their project pages: http://www.codeplex.com/MVPXML
I am developing a little app that retrieves an XML file, located on a remote server (http://example.com/myfile.xml)
This file is relatively big, and it contains a big list of geolocations with other information that I need to use for my app.
So I read this file remotely once and insert it into a little SqlCE file (database.sdf)
So If I need to be accessing geolocation #1, I ll just make a SELECT statement into this DATABASE instead of loading the whole XML file every time.
But I would like to know if its possible to do this without using .sdf files?
What is the most efficient way (fastest)?
Saving the big XML file once locally and load if every time I start my app to load it in a data set? this is would make the app a bit long to load every time
Saving the big XML file once locally and reading the nodes one by one to look for geolocation #1?
Or is it possible to retrieve geolocation #1 from the remote XML directly(http://example.com/myfile.xml) without reading the whole file?
Load the big XML file, convert it into an appropriate different data structure, save it to a file in an efficient format. (XML really isn't terribly efficient.)
I believe Marc Gravell's Protocol Buffers implementation works on the Compact Framework...
(None of the protobuf implementations are deemed production-ready yet, but a couple are close. We need testers!)
Re protobuf-net, there isn't a separate download for the CF version at the moment, but there is a csproj in the source for both CF 2.0 and CF 3.5.
To clarify on your question; actually protobuf-net doesn't even use a .proto file (at the moment); a .proto file just describes what the data is - protobuf-net simply looks at your classes and infers the schema from that (similar to how XmlSerializer / DataContractSerializer etc work). So there is not .proto - just the classes that look like your data.
However, before you embark on creating classes that look like your data, I wonder if you couldn't simply use GZIP or [PK]ZIP to compress the data, and transfer it "as is". XML generally compresses very well. Of course, finding a GZIP (etc) implementation for CF then becomes the issue.
Of course, if you want to use protobuf-net here, I'll happily advise etc if you get issues...
The other option is for your CF app to call into a web-service that has the data locally...
Why would you pull the entire file down to the CE device for this? It's a bandwidth waste and certainly doing the lookup on an embedded processor is going to be way slower than on the server regardless of storage format. You should have a service (Web, WCF or whatever) that allows you to ask it for the single geolocation you want.