Develop Reference

(C#) How to modify attribute's value in the existing XML file without loading or rewriting the whole file?

I'm making some huge XML files (several GB) with the help of XmlWriter and Linq2Xml.
This files are of type:
<Table recCount="" recLength="">
<Rec recId="1">..</Rec>
<Rec recId="2">..</Rec>
..
<Rec recId="n">..</Rec>
</Table>
I don't know values for Table's recCount and recLength attributes until I write all the inner Rec nodes, so I have to write values to these attributes at the very end.
Right now I'm writing all the inner Rec nodes to a temp file, calculate Table's attributes' values and write everything the way I've shown above to a resulting file. (copying everything from the temp file with all the Rec nodes)
I'm wondering if there is a way to modify these attributes' values without writing stuff to another file (like I do it right now) or loading the whole document into memory (which is obviously not possible due to size of these files)?

Heavily commented code. The basic idea is that in the first pass we write:
<?xml version="1.0" encoding="utf-8"?>
<Table recCount="$1" recLength="$2">
<!--Reserved space:++++++++++++++++-->
<Rec...
Then we go back to the beginning of the file and we rewrite the first three lines:
<?xml version="1.0" encoding="utf-8"?>
<Table recCount="1000" recLength="150">
<!--Reserved space:#############-->
The important "trick" here is that you can't "insert" into a file, you can only overwrite it. So we "reserve" some space for the digits (the Reserved space:#############. comment. There are many many ways we could have done it... For example, in the first pass we could have:
<Table recCount=" " recLength=" ">
and then (xml-legal but ugly):
<Table recCount="1000 " recLength="150 ">
Or we could have appended the space after the > of Table:
<Table recCount="" recLength="">
(there are 20 spaces after the >)
Then:
<Table recCount="1000" recLength="150">
(now there are are 13 spaces after the >)
Or we could have simply added the spaces without the <!-- --> on a new line...
The code:
int maxRecCountLength = 10; // int.MaxValue.ToString().Length
int maxRecLengthLength = 10; // int.MaxValue.ToString().Length
int tokenLength = 4; // 4 == $1 + $2, see below what $1 and $2 are
// Note that the reserved space will be in the form +++++++++++++++++++
string reservedSpace = new string('+', maxRecCountLength + maxRecLengthLength - tokenLength);
// You have to manually open the FileStream
using (var fs = new FileStream("out.xml", FileMode.Create))
// and add a StreamWriter on top of it
using (var sw = new StreamWriter(fs, Encoding.UTF8, 4096, true))
{
// Here you write on your StreamWriter however you want.
// Note that recCount and recLength have a placeholder $1 and $2.
int recCount = 0;
int maxRecLength = 0;
using (var xw = XmlWriter.Create(sw))
{
xw.WriteWhitespace("\r\n");
xw.WriteStartElement("Table");
xw.WriteAttributeString("recCount", "$1");
xw.WriteAttributeString("recLength", "$2");
// You have to add some white space that will be
// partially replaced by the recCount and recLength value
xw.WriteWhitespace("\r\n");
xw.WriteComment("Reserved space:" + reservedSpace);
// <--------- BEGIN YOUR CODE
for (int i = 0; i < 100; i++)
{
xw.WriteWhitespace("\r\n");
xw.WriteStartElement("Rec");
string str = string.Format("Some number: {0}", i);
if (str.Length > maxRecLength)
{
maxRecLength = str.Length;
}
xw.WriteValue(str);
recCount++;
xw.WriteEndElement();
}
// <--------- END YOUR CODE
xw.WriteWhitespace("\r\n");
xw.WriteEndElement();
}
sw.Flush();
// Now we read the first lines to modify them (normally we will
// read three lines, the xml header, the <Table element and the
// <-- Reserved space:
fs.Position = 0;
var lines = new List<string>();
using (var sr = new StreamReader(fs, sw.Encoding, false, 4096, true))
{
while (true)
{
string str = sr.ReadLine();
lines.Add(str);
if (str.StartsWith("<Table"))
{
// We read the next line, the comment line
str = sr.ReadLine();
lines.Add(str);
break;
}
}
}
string strCount = XmlConvert.ToString(recCount);
string strMaxRecLength = XmlConvert.ToString(maxRecLength);
// We do some replaces for the tokens
int oldLen = lines[lines.Count - 2].Length;
lines[lines.Count - 2] = lines[lines.Count - 2].Replace("=\"$1\"", string.Format("=\"{0}\"", strCount));
lines[lines.Count - 2] = lines[lines.Count - 2].Replace("=\"$2\"", string.Format("=\"{0}\"", strMaxRecLength));
int newLen = lines[lines.Count - 2].Length;
// Remove spaces from reserved whitespace
lines[lines.Count - 1] = lines[lines.Count - 1].Replace(":" + reservedSpace, ":" + new string('#', reservedSpace.Length - newLen + oldLen));
// We move back to just after the UTF8/UTF16 preamble
fs.Position = sw.Encoding.GetPreamble().Length;
// And we rewrite the lines
foreach (string str in lines)
{
sw.Write(str);
sw.Write("\r\n");
}
}
Slower .NET 3.5 way
In .NET 3.5 the StreamReader/StreamWriter want to close the base FileStream, so I have to reopen various times the file. This is a little little slower.
int maxRecCountLength = 10; // int.MaxValue.ToString().Length
int maxRecLengthLength = 10; // int.MaxValue.ToString().Length
int tokenLength = 4; // 4 == $1 + $2, see below what $1 and $2 are
// Note that the reserved space will be in the form +++++++++++++++++++
string reservedSpace = new string('+', maxRecCountLength + maxRecLengthLength - tokenLength);
string fileName = "out.xml";
int recCount = 0;
int maxRecLength = 0;
using (var sw = new StreamWriter(fileName))
{
// Here you write on your StreamWriter however you want.
// Note that recCount and recLength have a placeholder $1 and $2.
using (var xw = XmlWriter.Create(sw))
{
xw.WriteWhitespace("\r\n");
xw.WriteStartElement("Table");
xw.WriteAttributeString("recCount", "$1");
xw.WriteAttributeString("recLength", "$2");
// You have to add some white space that will be
// partially replaced by the recCount and recLength value
xw.WriteWhitespace("\r\n");
xw.WriteComment("Reserved space:" + reservedSpace);
// <--------- BEGIN YOUR CODE
for (int i = 0; i < 100; i++)
{
xw.WriteWhitespace("\r\n");
xw.WriteStartElement("Rec");
string str = string.Format("Some number: {0}", i);
if (str.Length > maxRecLength)
{
maxRecLength = str.Length;
}
xw.WriteValue(str);
recCount++;
xw.WriteEndElement();
}
// <--------- END YOUR CODE
xw.WriteWhitespace("\r\n");
xw.WriteEndElement();
}
}
var lines = new List<string>();
using (var sr = new StreamReader(fileName))
{
// Now we read the first lines to modify them (normally we will
// read three lines, the xml header, the <Table element and the
// <-- Reserved space:
while (true)
{
string str = sr.ReadLine();
lines.Add(str);
if (str.StartsWith("<Table"))
{
// We read the next line, the comment line
str = sr.ReadLine();
lines.Add(str);
break;
}
}
}
// We have to use the Stream overload of StreamWriter because
// we want to modify the text!
using (var fs = File.OpenWrite(fileName))
using (var sw = new StreamWriter(fs))
{
string strCount = XmlConvert.ToString(recCount);
string strMaxRecLength = XmlConvert.ToString(maxRecLength);
// We do some replaces for the tokens
int oldLen = lines[lines.Count - 2].Length;
lines[lines.Count - 2] = lines[lines.Count - 2].Replace("=\"$1\"", string.Format("=\"{0}\"", strCount));
lines[lines.Count - 2] = lines[lines.Count - 2].Replace("=\"$2\"", string.Format("=\"{0}\"", strMaxRecLength));
int newLen = lines[lines.Count - 2].Length;
// Remove spaces from reserved whitespace
lines[lines.Count - 1] = lines[lines.Count - 1].Replace(":" + reservedSpace, ":" + new string('#', reservedSpace.Length - newLen + oldLen));
// We move back to just after the UTF8/UTF16 preamble
sw.BaseStream.Position = sw.Encoding.GetPreamble().Length;
// And we rewrite the lines
foreach (string str in lines)
{
sw.Write(str);
sw.Write("\r\n");
}
}

Try to use the following approach.
You can set the default value to the attributes in the external xml schema.
When creating an xml document, you do not create these attributes. Here it is:
int count = 5;
int length = 42;
var writerSettings = new XmlWriterSettings { Indent = true };
using (var writer = XmlWriter.Create("data.xml", writerSettings))
{
writer.WriteStartElement("Table");
for (int i = 1; i <= count; i++)
{
writer.WriteStartElement("Rec");
writer.WriteAttributeString("recId", i.ToString());
writer.WriteString("..");
writer.WriteEndElement();
}
}
Thus, the xml looks like this:
<?xml version="1.0" encoding="utf-8"?>
<Table>
<Rec recId="1">..</Rec>
<Rec recId="2">..</Rec>
<Rec recId="3">..</Rec>
<Rec recId="4">..</Rec>
<Rec recId="5">..</Rec>
</Table>
Now create an xml schema for this document, which will specify the default values to the desired attributes.
string ns = "http://www.w3.org/2001/XMLSchema";
using (var writer = XmlWriter.Create("data.xsd", writerSettings))
{
writer.WriteStartElement("xs", "schema", ns);
writer.WriteStartElement("xs", "element", ns);
writer.WriteAttributeString("name", "Table");
writer.WriteStartElement("xs", "complexType", ns);
writer.WriteStartElement("xs", "sequence", ns);
writer.WriteStartElement("xs", "any", ns);
writer.WriteAttributeString("processContents", "skip");
writer.WriteAttributeString("maxOccurs", "unbounded");
writer.WriteEndElement();
writer.WriteEndElement();
writer.WriteStartElement("xs", "attribute", ns);
writer.WriteAttributeString("name", "recCount");
writer.WriteAttributeString("default", count.ToString()); // <--
writer.WriteEndElement();
writer.WriteStartElement("xs", "attribute", ns);
writer.WriteAttributeString("name", "recLength");
writer.WriteAttributeString("default", length.ToString()); // <--
writer.WriteEndElement();
}
Or much easier to create a schema as following:
XNamespace xs = "http://www.w3.org/2001/XMLSchema";
var schema = new XElement(xs + "schema",
new XElement(xs + "element", new XAttribute("name", "Table"),
new XElement(xs + "complexType",
new XElement(xs + "sequence",
new XElement(xs + "any",
new XAttribute("processContents", "skip"),
new XAttribute("maxOccurs", "unbounded")
)
),
new XElement(xs + "attribute",
new XAttribute("name", "recCount"),
new XAttribute("default", count) // <--
),
new XElement(xs + "attribute",
new XAttribute("name", "recLength"),
new XAttribute("default", length) // <--
)
)
)
);
schema.Save("data.xsd");
Please note the writing of the variables count and length - there should be your data.
The resulting schema will look like this:
<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="Table">
<xs:complexType>
<xs:sequence>
<xs:any processContents="skip" maxOccurs="unbounded" />
</xs:sequence>
<xs:attribute name="recCount" default="5" />
<xs:attribute name="recLength" default="42" />
</xs:complexType>
</xs:element>
</xs:schema>
Now, when reading an xml document, you must to add this schema - the default attribute values will be taken from it.
XElement xml;
var readerSettings = new XmlReaderSettings();
readerSettings.ValidationType = ValidationType.Schema; // <--
readerSettings.Schemas.Add("", "data.xsd"); // <--
using (var reader = XmlReader.Create("data.xml", readerSettings)) // <--
{
xml = XElement.Load(reader);
}
xml.Save(Console.Out);
Console.WriteLine();
The result:
<Table recCount="5" recLength="42">
<Rec recId="1">..</Rec>
<Rec recId="2">..</Rec>
<Rec recId="3">..</Rec>
<Rec recId="4">..</Rec>
<Rec recId="5">..</Rec>
</Table>

You could try to load the xml file into a dataset as it would be easier to compute your attributes that way. Plus the memory management is done by the DataSet layer. Why not give it a try and let us all know too of the results.

I think the FileStream Class will be helpful to you. Take a look at the Read and Write methods.

change value of XML element

I have XML which is in project folder. I am loading the contents using this
XmlDocument doc = new XmlDocument();
string testFilesLocation = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
string dataSource;
string xmlFileName = "Claim.txt";
if (System.IO.Directory.Exists(testFilesLocation + #"\Resources"))
{
dataSource = testFilesLocation + #"\Resources\" + xmlFileName;
}
else
{
dataSource = testFilesLocation + #"\" + xmlFileName;
}
doc.Load(dataSource);
XML has following nodes
<ClaimKeyInfo>
<CompanyID>XXXX</CompanyID>
<ClaimNum>XX-XXXXX-XX<ClaimNum>
</ClaimKeyInfo>
<ClaimInfo>
<ClaimNum>XX-XXXXX-XX</ClaimNum>
<ClaimSensitivityInd>N</ClaimSensitivityInd>
<ClaimStatus>Open</ClaimStatus>
<ClaimInfo>
I am doing this to get ClaimNum elements
XmlElement root = doc.DocumentElement;
XmlNodeList elemList = root.GetElementsByTagName("ClaimNum");
for (int i = 0; i< elemList.Count; i++)
{
elemList[i].InnerXml = "YY-YYYYY-YY";
doc.Save(dataSource);
}
I do get both the elements in elemList but I am not able to change the values inside it.
Any help would be appreciated.

You can use LINQ to XML for that.
var xDoc = XDocument.Load("filePath");
var numbers = xDoc.Descendants("ClaimNum");
foreach(var number in numbers)
{
// do your work with numbers
number.Value = "123456";
}
xDoc.Save("filePath");
Here, the numbers are contains XElements which is your ClaimNums.You can change the Value property and save the XML file.If you want to get specific numbers you can use Where extension method.If you want more details see these documentations:
How to: Find an Element with a Specific Attribute
How to: Find an Element with a Specific Child Element

how to read and change the xml file in C#

Thank you very much for reading my question.
and this is my xml file. (for node Songs, many childNodes named Song)
<?xml version="1.0" encoding="utf-8" ?>
<xmlData>
<version>1.0</version>
<Songs>
<Song>
<artist>mic</artist>
<track>2</track>
<column>happy</column>
<date>14</date>
</Song>
<Song>
<artist>cool</artist>
<track>2</track>
<column>work</column>
<date>4</date>
</Song>
</Songs>
</xmlData>
reading xml, i use the following code:
XmlDocument doc = new XmlDocument();
doc.Load(xmlFilePath);
XmlNode versionNode = doc.SelectSingleNode(#"/xmlData/version");
Console.WriteLine(versionNode.Name + ":\t" + versionNode.InnerText);
XmlNode SongsNode = doc.SelectSingleNode(#"/xmlData/Songs");
Console.WriteLine(SongsNode.Name + "\n");
XmlDocument docSub = new XmlDocument();
docSub.LoadXml(SongsNode.OuterXml);
XmlNodeList SongList = docSub.SelectNodes(#"/Songs/Song");
if (SongList != null)
{
foreach (XmlNode SongNode in SongList)
{
XmlNode artistDetail = SongNode.SelectSingleNode("artist");
Console.WriteLine(artistDetail.Name + "\t: " + artistDetail.InnerText);
XmlNode trackDetail = SongNode.SelectSingleNode("track");
Console.WriteLine(trackDetail.Name + "\t: " + trackDetail.InnerText);
XmlNode columnDetail = SongNode.SelectSingleNode("column");
Console.WriteLine(columnDetail.Name + "\t: " + columnDetail.InnerText);
XmlNode dateDetail = SongNode.SelectSingleNode("date");
Console.WriteLine(dateDetail.Name + "\t: " + dateDetail.InnerText + "\n");
}
}
it seems working.
but how can i write the change to xml file?
maybe, i will change some childNode in Song, and may delete the whole chindNode by artist keyword.
is it possible such as this function
bool DeleteSongByArtist(string sArtist);
bool ChangeNodeInSong(string sArtist, string sNodeName, string value);
because the "Reading solution is "XmlDucoment", so it is better if "changing solution" by using "XmlDocument"
but, if you have better idea to read and change the xml file, please give me the sample code... and please don't write a name of solution such as "Ling to xml"...acutally, i do many testes, but failed.

Welcome to Stackoverflow!
You can change the nodes simply by setting a new .Value or in your case .InnerText.
Sample
// change the node
trackDetail.InnerText = "NewValue"
// save the document
doc.Save(xmlFilePath);
More Information
How To: Modify an Existing Xml File
MSDN - XmlDocument.Save Method

You need to use an XmlWriter. The easiest way to do it would be something like this...
using(XmlWriter writer = new XmlWriter(textWriter))
{
doc.WriteTo(writer);
}
Where textWriter is your initialized Text Writer.
Actually, forget that... the easiest way is to call...
doc.Save(xmlFilePath);
To delete an artist by artist name add the following method:
bool DeleteSongByArtist(XmlDocument doc, string artistName)
{
XmlNodeList SongList = doc.SelectNodes(#"/Songs/Song");
if (SongList != null)
{
for (int i = SongList.Count - 1; i >= 0; i--)
{
if (SongList[i]["artist"].InnerText == artistName && SongList[i].ParentNode != null)
{
SongList[i].ParentNode.RemoveChild(SongList[i]);
}
}
}
}
You probably want to clean it up a bit more to be more resilient. When you call it, change your initial code to be like this. Don't create the subDocument as you want to work with the entire XmlDocument.
XmlDocument doc = new XmlDocument();
doc.Load(xmlFilePath);
XmlNode versionNode = doc.SelectSingleNode(#"/xmlData/version");
Console.WriteLine(versionNode.Name + ":\t" + versionNode.InnerText);
XmlNode SongsNode = doc.SelectSingleNode(#"/xmlData/Songs");
Console.WriteLine(SongsNode.Name + "\n");
XmlNodeList SongList = doc.SelectNodes(#"/Songs/Song");
if (SongList != null)
{
foreach (XmlNode SongNode in SongList)
{
XmlNode artistDetail = SongNode.SelectSingleNode("artist");
Console.WriteLine(artistDetail.Name + "\t: " + artistDetail.InnerText);
XmlNode trackDetail = SongNode.SelectSingleNode("track");
Console.WriteLine(trackDetail.Name + "\t: " + trackDetail.InnerText);
XmlNode columnDetail = SongNode.SelectSingleNode("column");
Console.WriteLine(columnDetail.Name + "\t: " + columnDetail.InnerText);
XmlNode dateDetail = SongNode.SelectSingleNode("date");
Console.WriteLine(dateDetail.Name + "\t: " + dateDetail.InnerText + "\n");
}
}

You aren't able to save your changes because you made changes to an entirely new document!
You likely meant to do the following:
XmlNode SongsNode = doc.SelectSingleNode(#"/xmlData/Songs");
Console.WriteLine(SongsNode.Name + "\n");
// Don't make a new XmlDocument here! Use your existing one
XmlNodeList SongList = SongsNode.SelectNodes(#"/Song");
At this point SongList is still living inside doc. Now when you call:
doc.Save(xmlFilePath);
Your changes will be saved as you intended.
If you're looking to delete nodes that match certain criteria:
// Use XPath to find the matching node
XmlNode song = SongsNode.SelectSingleNode(#"/Song[artist='" + artist + "']");
// Remove it from its Parent
SongsNode.RemoveChild(song);
If you're looking to add a new node:
// Create the new nodes using doc
XmlNode newSong = doc.CreateElement("Song");
XmlNode artist = doc.CreateElement("artist");
artist.InnerText = "Hello";
// Begin the painstaking process of creation/appending
newSong.AppendChild(artist);
// rinse...repeat...
// Finally add the new song to the SongsNode
SongsNode.AppendChild(newSong);

You could do
XmlNodeList SongList = doc.SelectNodes(#"//Songs/Song");
The // tells it to select the Songs node anywhere in document. This is better than
doc.SelectNodes(#"/document/level1/music/Songs")
Note that the above statement is oviously not for your xml, but to prove a point about //
Using // removes the need for your docSub document and SongsNode element.
To add then delete a song, just use the following
XmlDocument doc = new XmlDocument();
XmlElement ea = doc.SelectSingleNode("//songs");
XmlElement el = doc.CreateElement("song");
XmlElement er;
ea.AppendChild(el);
//doing my work with ea
//you could use innerxml.
el.InnerXml = "<artist>Judas Priest</artist><track>7</track><column>good</column><date>1</date>";
//or you can treat each node as above
er = doc.CreateElement("Name");
el.AppendChild(er);
er.InnerText = "The Ripper";
//but you don't nead this song any more?
ea.RemoveChild(el);
//so it's gone.
And thats all there is to it.

please don't write a name of solution such as "Ling to xml"...acutally, i do many testes, but failed.
Still I think, this is a very good time to start to use Linq2Xml. If you don't like it, just ignore.
XDocument xDoc = XDocument.Load(new StringReader(xml));
//Load Songs
var songs = xDoc.Descendants("Song")
.Select(s => new
{
Artist = s.Element("artist").Value,
Track = s.Element("track").Value,
Column = s.Element("column").Value,
Date = s.Element("date").Value,
})
.ToArray();
//Delete Songs
string songByArtist="mic";
xDoc.Descendants("Song")
.Where(s => s.Element("artist").Value == songByArtist)
.Remove();
string newXml = xDoc.ToString();

SqlCommand Query and iteration to execute data into multiple XML files with my given program

I am writing a prototype to programmatically get video data from a database and use C# to place this data into an XML manifest file. Each video is an asset element inside XML, and I am querying all the data I need from a SQLCommand.
Problem: The XML file can only hold up to 100 video assets per file, and so I need to come up with an iterator that saves 100 assets per file until I reach the end row in the database. I am using while(r.Read()), a SqlDataReader to go into the data base using a SqlCommand.
I would like to know how to go about this process of appending 100 assets per file using a certain SQLCommand and an iteration inside the reader to process all files needed.
Below is my code thus far! (Clearly I am going to have to change some things around, such as the global elements of the XML file, which need to go in every XML file created)
protected void btnExecute_Click(object sender, EventArgs e)
{
//On btn click, call method to execute program and save all files in local path
GetVideoData();
}
protected void GetVideoData()
{
string preparer = "AgencyOasis";
string type = "VIDEO_FULL";
XmlDocument doc = new XmlDocument();
XmlNode docNode = doc.CreateXmlDeclaration("1.0", "UTF-8", null);
doc.AppendChild(docNode);
//add global elements:
//create parameters for each attribute in root to be replaced, and append the new root tag to empty XML doc
XmlElement root = doc.CreateElement("publisher-upload-manifest");
root.SetAttribute("publisher-id", tbPublisherID.Text);
root.SetAttribute("preparer", preparer);
doc.AppendChild(root);
//append notify as child to root, already in doc
if (!string.IsNullOrEmpty(tbEmail.Text)) {
XmlElement notify = doc.CreateElement("notify");
notify.SetAttribute("email", tbEmail.Text);
root.AppendChild(notify);
}
//THE REST OF THE ELEMENTS ARE A UNIQUE CASE FOR EACH VIDEO, THEREFORE SHOULD LOOP INSIDE THE QUERY RESULT SET, PER VIDEO.
string sql100 = "SELECT TOP 100 Location, VideoLibraryId, Title, Keywords, IsActive, Description FROM VideoLibrary";
const string _connStringName = "SanfordVideosConnectionString";
string dsn = ConfigurationManager.ConnectionStrings[_connStringName].ConnectionString;
using (SqlConnection conn = new SqlConnection(dsn))
using (SqlCommand cmd = new SqlCommand(sql100, conn))
{
conn.Open();
SqlDataReader r = cmd.ExecuteReader();
while (r.Read())
{
//while going through each row with above SQL command, set data to element attributes in doc for each asset
XmlElement asset = doc.CreateElement("asset");
asset.SetAttribute("filename", r["Location"].ToString());
asset.SetAttribute("refid", r["VideoLibraryId"].ToString());
// TODO: NEED ACTUAL FILE LOCATION BEFORE I CAN EXTRACT THE SIZE OF THE FILE ITSELF
/*create new FileInfo object to get length of existing video file
FileInfo f = new FileInfo(r["Location"].ToString());
long f1 = f.Length; */
//asset.SetAttribute("size", "10"); // TODO: f1.ToString() for static value of 2nd #
// TODO: NEED ACTUAL FILE LOCATION AGAIN FOR HASH-CODE
//asset.SetAttribute("hash-code", "10"); //TODO: GetMD5Hash(r["Location"].ToString())
//setting the type globally for all videos to VIDEO_FULL ensures FLV and MP4 formats
asset.SetAttribute("type", type);
root.AppendChild(asset);
XmlElement title = doc.CreateElement("title");
title.SetAttribute("name", r["Title"].ToString());
title.SetAttribute("refid", r["VideoLibraryId"].ToString());
title.SetAttribute("active", r["IsActive"].ToString().ToUpper());
// TODO: CHECK TO SEE IF VIDEO-FULL-REFID IS CORRECT
//title.SetAttribute("video-full-refid", r["Location"].ToString() + "-" + r["VideoLibraryId"].ToString());
XmlElement shortDesc = doc.CreateElement("short-description");
shortDesc.InnerText = GetTrimmedDescription(250, r["Description"].ToString());
title.AppendChild(shortDesc);
root.AppendChild(title);
XmlElement longDesc = doc.CreateElement("long-description");
longDesc.InnerText = GetTrimmedDescription(5000, r["Description"].ToString());
title.AppendChild(longDesc);
root.AppendChild(title);
}
}
//TEMPORARY FILE SAVE LOCATION TODO: SAVE MULTIPLE FILES IN LOCAL FOLDER
//returns the directory from where the current application domain was loaded
//string xmlPath = System.IO.Path.Combine(AppDomain.CurrentDomain.BaseDirectory, inputFileName1);
string xmlPath = Server.MapPath(txtBoxInput.Text);
XmlTextWriter writer = new XmlTextWriter(xmlPath, null);
writer.Formatting = Formatting.Indented;
doc.Save(xmlPath);
}
//Trims long and short descriptions to max size of chars depending on max size (250 for short and 5000 for long)
public string GetTrimmedDescription(int maxLength, string desc) {
if (desc.Length > maxLength)
{
return desc.Substring(0, (maxLength - 4)) + " ...";
}
else
{
return desc;
}
}
Feel free to ask me any questions about the program and I'll try my best to explain!

If I keep your code mostly as is, here is a possible solution. I have added in some comments where I refactored pieces.
The general idea is to fetch everything (not just 100 records) and introduce a counter that tracks where you are at. Every 100 records you spit out the file and reset. After you've read everything you have to spit out what is remaining. Since there are multiple files created, you will also have to track that and use a different filename. I just added a file suffix. I think the code below should work for you.
protected void btnExecute_Click(object sender, EventArgs e)
{
//On btn click, call method to execute program and save all files in local path
GetVideoData();
}
protected void GetVideoData()
{
string type = "VIDEO_FULL";
XmlDocument doc;
XmlElement root;
// Refactor document creation to its own function
doc = CreateANewDoc();
// Refactor root creation to its own function
root = CreateANewRoot(doc);
//THE REST OF THE ELEMENTS ARE A UNIQUE CASE FOR EACH VIDEO, THEREFORE SHOULD LOOP INSIDE THE QUERY RESULT SET, PER VIDEO.
// remove the top 100 and fetch everything
string sql100 = "SELECT Location, VideoLibraryId, Title, Keywords, IsActive, Description FROM VideoLibrary";
const string _connStringName = "SanfordVideosConnectionString";
string dsn = ConfigurationManager.ConnectionStrings[_connStringName].ConnectionString;
using (SqlConnection conn = new SqlConnection(dsn))
using (SqlCommand cmd = new SqlCommand(sql100, conn))
{
conn.Open();
SqlDataReader r = cmd.ExecuteReader();
// declare some counters
int counter = 0;
int filecounter = 0;
while (r.Read())
{
counter++; // Increment counter
// when you hit 100, write stuff out and reset
if (counter == 100 )
{
filecounter++; // Increment filecounter
DumpOutFile(doc, filecounter);
// Reset counter
counter = 0;
// Reset doc
doc = CreateANewDoc();
root = CreateANewRoot(doc);
}
//while going through each row with above SQL command, set data to element attributes in doc for each asset
XmlElement asset = doc.CreateElement("asset");
asset.SetAttribute("filename", r["Location"].ToString());
asset.SetAttribute("refid", r["VideoLibraryId"].ToString());
// TODO: NEED ACTUAL FILE LOCATION BEFORE I CAN EXTRACT THE SIZE OF THE FILE ITSELF
/*create new FileInfo object to get length of existing video file
FileInfo f = new FileInfo(r["Location"].ToString());
long f1 = f.Length; */
//asset.SetAttribute("size", "10"); // TODO: f1.ToString() for static value of 2nd #
// TODO: NEED ACTUAL FILE LOCATION AGAIN FOR HASH-CODE
//asset.SetAttribute("hash-code", "10"); //TODO: GetMD5Hash(r["Location"].ToString())
//setting the type globally for all videos to VIDEO_FULL ensures FLV and MP4 formats
asset.SetAttribute("type", type);
root.AppendChild(asset);
XmlElement title = doc.CreateElement("title");
title.SetAttribute("name", r["Title"].ToString());
title.SetAttribute("refid", r["VideoLibraryId"].ToString());
title.SetAttribute("active", r["IsActive"].ToString().ToUpper());
// TODO: CHECK TO SEE IF VIDEO-FULL-REFID IS CORRECT
//title.SetAttribute("video-full-refid", r["Location"].ToString() + "-" + r["VideoLibraryId"].ToString());
XmlElement shortDesc = doc.CreateElement("short-description");
shortDesc.InnerText = GetTrimmedDescription(250, r["Description"].ToString());
title.AppendChild(shortDesc);
root.AppendChild(title);
XmlElement longDesc = doc.CreateElement("long-description");
longDesc.InnerText = GetTrimmedDescription(5000, r["Description"].ToString());
title.AppendChild(longDesc);
root.AppendChild(title);
}
// Dump out whatever is left (handles the remainder after division by 100
DumpOutFile(doc, filecounter + 1);
}
}
//Trims long and short descriptions to max size of chars depending on max size (250 for short and 5000 for long)
public string GetTrimmedDescription(int maxLength, string desc) {
if (desc.Length > maxLength)
{
return desc.Substring(0, (maxLength - 4)) + " ...";
}
else
{
return desc;
}
}
private XmlDocument CreateANewDoc()
{
XmlDocument doc = new XmlDocument();
XmlNode docNode = doc.CreateXmlDeclaration("1.0", "UTF-8", null);
doc.AppendChild(docNode);
return doc;
}
private XmlElement CreateANewRoot(XmlDocument doc)
{
string preparer = "AgencyOasis";
//add global elements:
//create parameters for each attribute in root to be replaced, and append the new root tag to empty XML doc
XmlElement root = doc.CreateElement("publisher-upload-manifest");
//root.SetAttribute("publisher-id", tbPublisherID.Text);
root.SetAttribute("publisher-id", "tbPublisherID.Text");
root.SetAttribute("preparer", preparer);
doc.AppendChild(root);
//append notify as child to root, already in doc
if (!string.IsNullOrEmpty(tbEmail.Text)) {
XmlElement notify = doc.CreateElement("notify");
notify.SetAttribute("email", tbEmail.Text);
root.AppendChild(notify);
}
return root;
}
private void DumpOutFile(XmlDocument doc, int filenumber)
{
//TEMPORARY FILE SAVE LOCATION TODO: SAVE MULTIPLE FILES IN LOCAL FOLDER
//returns the directory from where the current application domain was loaded
//string xmlPath = System.IO.Path.Combine(AppDomain.CurrentDomain.BaseDirectory, inputFileName1);
string xmlPath = Server.MapPath(txtBoxInput.Text + filenumber.ToString());
XmlTextWriter writer = new XmlTextWriter(xmlPath, null);
writer.Formatting = Formatting.Indented;
doc.Save(xmlPath);
}

How do I parse a word between two elements?and extract it

example
<reference>employee</reference>
<data>123</data>
how to parse using c# so that i should get employee and 123 as output

You can make an XML document out of it, and parse it:
string info = "<reference>employee</reference><data>123</data>";
XmlDocument doc = new XmlDocument();
doc.LoadXml("<root>" + info + "</root>");
string reference = doc.DocumentElement.FirstChild.InnerText;
string data = doc.DocumentElement.FirstChild.NextSibling.InnerText;
Another option is to use a regular expression to parse the string:
string info = "<reference>employee</reference><data>123</data>";
MatchCollection m = Regex.Matches(info, "<.+?>(.+?)</.+?>");
string reference = m[0].Groups[1].Value;
string data = m[1].Groups[1].Value;
Or simple string manipulation:
string info = "<reference>employee</reference><data>123</data>";
int start = info.IndexOf("<reference>") + 11;
string reference = info.Substring(start, info.IndexOf('<', start) - start);
start = info.IndexOf("<data>") + 6;
string data = info.Substring(start, info.IndexOf('<', start) - start);

string xml = #"<root>
<reference>employee</reference>
<data>123</data>
</root>";
System.Xml.XmlDocument doc = new System.Xml.XmlDocument();
doc.LoadXml(xml);
string employee = doc.SelectSingleNode("reference").InnerText;
string data = doc.SelectSingleNode("data").InnerText;