Develop Reference

XmlSerializer.Serialize BOM missing

I am using this code to store my class:
FileStream stream = new FileStream(myPath, FileMode.Create);
XmlSerializer serializer = new XmlSerializer(typeof(myClass));
serializer.Serialize(stream, myClass);
stream.Close();
This writes a file that I can read alright with XmlSerializer.Deserialize. The generated file, however, is not a proper text file. XmlSerializer.Serialize doesn't store a BOM, but still inserts multibyte characters. Thus it is implicitely declared an ANSI file (because we expect an XML file to be a text file, and a text file without a BOM is considered ANSI by Windows), showing ö as ö in some editors.
Is this a known bug? Or some setting that I'm missing?
Here is what the generated file starts with:
<?xml version="1.0"?>
<SvnProjects xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
The first byte in the file is hex 3C, i.e the <.

Having or not having a BOM is not a definition of a "proper text file". In fact, I'd say that the most typical format these days is UTF-8 without BOM; I don't think I've ever seen anyone actually use the UTF-8 BOM in real systems! But: if you want a BOM, that's fine: just pass the correct Encoding in; if you want UTF-8 with BOM:
using (var writer = XmlWriter.Create(myPath, s_settings))
{
XmlSerializer serializer = new XmlSerializer(typeof(MyClass));
serializer.Serialize(writer, obj);
}
with:
static readonly XmlWriterSettings s_settings =
new XmlWriterSettings { Encoding = new UTF8Encoding(true) };
The result of this is a file that starts EF-BB-BF, the UTF-8 BOM.
If you want a different encoding, then just replace new UTF8Encoding with whatever you did want, remembering to enable the BOM.
(note: the static Encoding.UTF8 instance has the BOM enabled, but IMO it is better to be very explicit here if you specifically intend to use a BOM, just like you should be very explicit about what Encoding you intended to use)
Edit: the key difference here is that Serialize(Stream, object) ends up using:
XmlTextWriter xmlWriter = new XmlTextWriter(stream, encoding: null) {
Formatting = Formatting.Indented,
Indentation = 2
};
which then ends up using:
public StreamWriter(Stream stream) : this(stream,
encoding: UTF8NoBOM, // <==== THIS IS THE PROBLEM
bufferSize: 1024, leaveOpen: false)
{
}
so: UTF-8 without BOM is the default if you use that API.

you must xml an instance not a class definition
for getting Unicode you must declare a XmlWriter or TextWriter
FileStream stream = new FileStream(myPath, FileMode.Create);
XmlSerializer serializer = new XmlSerializer(typeof(myClass));
XmlWriter writer = new XmlTextWriter(fs, Encoding.Unicode);
serializer.Serialize(writer, myClass);
stream.Close();

xslt always gives me UTF-16 with slashes

I have the following bit of code in C# to convert an XML file to another using XSLT/
string xmlInput = #"<?xml version='1.0' encoding='UTF-8'?><catalog><cd><title> Empire Burlesque </title ><artist> Bob Dylan </artist><country> USA </country><company> Columbia </company><price> 10.90 </price><year> 1985 </year></cd></catalog>";
///////////////////////////////////////////////////////////////
string xmlOutput = String.Empty;
using (StringReader sri = new StringReader(xmlInput))
{
using (XmlReader xri = XmlReader.Create(sri))
{
XslCompiledTransform xslt = new XslCompiledTransform();
//xslt.Load(xrt);
xslt.Load(#"XSLT/slide2.xslt");
using (StringWriter sw = new StringWriter())
using (XmlWriter xwo = XmlWriter.Create(sw, new XmlWriterSettings { Encoding = Encoding.UTF8 }))
{
xslt.Transform(xri, xwo);
xmlOutput = sw.ToString();
}
}
}
xmlOutput gives me "<?xml version=\"1.0\" encoding=\"utf-16\"?><root> Empire Burlesque </root>"
How can I get utf-8 and no slashes?

.NET strings are sequences of UTF-16 encoded characters and StringWriter/StringBuilder default to that encoding. (source https://forums.asp.net/post/3240311.aspx)
So you need to make a class which inherits of the default stringwriter:
public class StringWriterWithEncoding : StringWriter
{
Encoding myEncoding;
public override Encoding Encoding
{
get
{
return myEncoding;
}
}
public StringWriterWithEncoding(Encoding encoding) : base()
{
myEncoding = encoding;
}
public StringWriterWithEncoding(Encoding encoding) : base(CultureInfo.CurrentCulture)
{
myEncoding = encoding;
}
public StringWriterWithEncoding(StringBuilder sb, Encoding encoding) : base(sb, CultureInfo.CurrentCulture)
{
myEncoding = encoding;
}
}
and create an instance of that e.g. StringWriterWithEncoding utf8Writer = new StringWriterWithEncoding(Encoding.UTF8); and pass that as the third argument to the Transform method of your XslCompiledTransform.
use like this:
StringBuilder sb = new StringBuilder();
using (StringWriterWithEncoding sw = new StringWriterWithEncoding(sb, Encoding.UTF8))
{
XslCompiledTransform xslt = new XslCompiledTransform();
xslt.Load(#"XSLT/slide2.xslt");
xslt.Transform(xri, sw);
}
xmlOutput = sb.ToString();

First issue is caused by StringWriter
using (StringWriter sw = new StringWriter())
using (XmlWriter xwo = XmlWriter.Create(sw, new XmlWriterSettings { Encoding = Encoding.UTF8 }))
Even though you specifically set XmlWriterSettings.Encoding to UTF-8, you specify output stream to be StringWriter and since .NET strings are UTF-16, XmlWriter is forced to use UTF-16.
If you use for instance FileStream instead of StringWriter, output will be in UTF-8 or whatever encoding you specify.
Slashes issue is just your IDE escaping it. If you print xmlOutput to Console you will see it contains no extra slashes.

You can include this line in your XSLT stylesheet:
<xsl:output encoding="utf-8"/>
(or of course whichever encoding you prefer), and it will automatically set the output settings to the utf-8 encoding.

I believe that using the MemoryStream is a better way to handle this. .net strings are utf-16 internally and that's how they are encoded when you write to a StringWriter of StringBuilder object. Using a memory stream, you avoid that pitfall.
string xmlDoc = "";
// Use a memory stream to avoid the .net internal string utf-16 encoding pitfall.
using (MemoryStream xmlStream = new MemoryStream())
using (XmlReader xmlReader = XmlReader.Create(new StringReader(xmlAsText)))
using (XmlReader xsltReader = XmlReader.Create(new StringReader(xsltAsText)))
{
// Transform XML string to new XML based on the XSLT
// Load the XSLT and transform source XML to target XML
XslCompiledTransform myXslTrans = new XslCompiledTransform();
myXslTrans.Load(xsltReader);
myXslTrans.Transform(xmlReader, null, xmlStream);
// Using the encoding from the xslt, transform the xml stream bytes to the xml string.
// If no encoding in xslt, defaults to UTF-8.
xmlDoc = myXslTrans.OutputSettings.Encoding.GetString(xmlStream.ToArray());
// Remove the BOM if it exists
string byteOrderMark = myXslTrans.OutputSettings.Encoding.GetString(myXslTrans.OutputSettings.Encoding.GetPreamble());
if (xmlDoc.StartsWith(byteOrderMark, StringComparison.Ordinal))
{
xmlDoc = xmlDoc.Remove(0, byteOrderMark.Length);
}
}
If there is no encoding attribute in the stylesheet, then you get UTF-8 by default, not UTF-16. This is the same as if you were to write it directly to file. I can't say how it works for different cultures, sorry.
I'm suggesting this method over the other methods where they are hard coding UTF-8. This works with whatever (valid) encoding is in the style sheet, such as ISO-8859-1.

How to return xml as UTF-8 instead of UTF-16

I am using a routine that serializes <T>. It works, but when downloaded to the browser I see a blank page. I can view the page source or open the download in a text editor and I see the xml, but it is in UTF-16 which I think is why browser pages show blank?
How do I modify my serializer routine to return UTF-8 instead of UTF-16?
The XML source returned:
<?xml version="1.0" encoding="utf-16"?>
<ArrayOfString xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<string>January</string>
<string>February</string>
<string>March</string>
<string>April</string>
<string>May</string>
<string>June</string>
<string>July</string>
<string>August</string>
<string>September</string>
<string>October</string>
<string>November</string>
<string>December</string>
<string />
</ArrayOfString>
An example call to the serializer:
DateTimeFormatInfo dateTimeFormatInfo = new DateTimeFormatInfo();
var months = dateTimeFormatInfo.MonthNames.ToList();
string SelectionId = "1234567890";
return new XmlResult<List<string>>(SelectionId)
{
Data = months
};
The Serializer:
public class XmlResult<T> : ActionResult
{
private string filename = DateTime.Now.ToString("ddmmyyyyhhss");
public T Data { private get; set; }
public XmlResult(string selectionId = "")
{
if (selectionId != "")
{
filename = selectionId;
}
}
public override void ExecuteResult(ControllerContext context)
{
HttpContextBase httpContextBase = context.HttpContext;
httpContextBase.Response.Buffer = true;
httpContextBase.Response.Clear();
httpContextBase.Response.AddHeader("content-disposition", "attachment; filename=" + filename + ".xml");
httpContextBase.Response.ContentType = "text/xml";
using (StringWriter writer = new StringWriter())
{
XmlSerializer xml = new XmlSerializer(typeof(T));
xml.Serialize(writer, Data);
httpContextBase.Response.Write(writer);
}
}
}

You can use a StringWriter that will force UTF8. Here is one way to do it:
public class Utf8StringWriter : StringWriter
{
// Use UTF8 encoding but write no BOM to the wire
public override Encoding Encoding
{
get { return new UTF8Encoding(false); } // in real code I'll cache this encoding.
}
}
and then use the Utf8StringWriter writer in your code.
using (StringWriter writer = new Utf8StringWriter())
{
XmlSerializer xml = new XmlSerializer(typeof(T));
xml.Serialize(writer, Data);
httpContextBase.Response.Write(writer);
}
answer is inspired by Serializing an object as UTF-8 XML in .NET

Encoding of the Response
I am not quite familiar with this part of the framework. But according to the MSDN you can set the content encoding of an HttpResponse like this:
httpContextBase.Response.ContentEncoding = Encoding.UTF8;
Encoding as seen by the XmlSerializer
After reading your question again I see that this is the tough part. The problem lies within the use of the StringWriter. Because .NET Strings are always stored as UTF-16 (citation needed ^^) the StringWriter returns this as its encoding. Thus the XmlSerializer writes the XML-Declaration as
<?xml version="1.0" encoding="utf-16"?>
To work around that you can write into an MemoryStream like this:
using (MemoryStream stream = new MemoryStream())
using (StreamWriter writer = new StreamWriter(stream, Encoding.UTF8))
{
XmlSerializer xml = new XmlSerializer(typeof(T));
xml.Serialize(writer, Data);
// I am not 100% sure if this can be optimized
httpContextBase.Response.BinaryWrite(stream.ToArray());
}
Other approaches
Another edit: I just noticed this SO answer linked by jtm001. Condensed the solution there is to provide the XmlSerializer with a custom XmlWriter that is configured to use UTF8 as encoding.
Athari proposes to derive from the StringWriter and advertise the encoding as UTF8.
To my understanding both solutions should work as well. I think the take-away here is that you will need one kind of boilerplate code or another...

To serialize as UTF8 string:
private string Serialize(MyData data)
{
XmlSerializer ser = new XmlSerializer(typeof(MyData));
// Using a MemoryStream to store the serialized string as a byte array,
// which is "encoding-agnostic"
using (MemoryStream ms = new MemoryStream())
// Few options here, but remember to use a signature that allows you to
// specify the encoding
using (XmlTextWriter tw = new XmlTextWriter(ms, Encoding.UTF8))
{
tw.Formatting = Formatting.Indented;
ser.Serialize(tw, data);
// Now we get the serialized data as a string in the desired encoding
return Encoding.UTF8.GetString(ms.ToArray());
}
}
To return it as XML on a web response, don't forget to set the response encoding:
string xml = Serialize(data);
Response.ContentType = "application/xml";
Response.ContentEncoding = System.Text.Encoding.UTF8;
Response.Output.Write(xml);

How to write System.Xml.Linq.XElement using XmlWriter to a stream

I have an XElement instance and I wish to write to a stream using XmlWriter class. Why? Well, one of the configuration settings defines whether to use binary Xml or not. Based on this setting a suitable XmlWriter instance is created - either by XmlWriter.Create(stream) or XmlDictionaryWriter.CreateBinaryWriter(stream)).
Anyway, I am trying the following code, but it leaves the stream empty:
using (var stream = new MemoryStream())
{
var xmlReader = new XDocument(xml).CreateReader();
xmlReader.MoveToContent();
var xmlWriter = GetXmlWriter(stream);
xmlWriter.WriteNode(xmlReader, true);
return stream.ToArray();
}
I have checked, xmlReader is properly aligned after MoveToContent at the root XML element.
I must be doing something wrong, but what?
Thanks.

You haven't shown what GetXmlWriter does... but have you tried just flushing the writer?
xmlWriter.Flush();
Alternatively, wrap the XmlWriter in another using statement:
using (var stream = new MemoryStream())
{
var xmlReader = new XDocument(xml).CreateReader();
xmlReader.MoveToContent();
using (var xmlWriter = GetXmlWriter(stream))
{
xmlWriter.WriteNode(xmlReader, true);
}
return stream.ToArray();
}
You might want to do the same for the XmlReader as well, although in this particular case I don't believe you particularly need to.
Having said all this, I'm not entirely sure why you're using an XmlReader at all. Any reason you can't just find the relevant XElement and use XElement.WriteTo(XmlWriter)? Or if you're trying to copy the whole document, just use XDocument.WriteTo(XmlWriter)

Writing XML files using XmlTextWriter with ISO-8859-1 encoding

I'm having a problem writing Norwegian characters into an XML file using C#. I have a string variable containing some Norwegian text (with letters like æøå).
I'm writing the XML using an XmlTextWriter, writing the contents to a MemoryStream like this:
MemoryStream stream = new MemoryStream();
XmlTextWriter xmlTextWriter = new XmlTextWriter(stream, Encoding.GetEncoding("ISO-8859-1"));
xmlTextWriter.Formatting = Formatting.Indented;
xmlTextWriter.WriteStartDocument(); //Start doc
Then I add my Norwegian text like this:
xmlTextWriter.WriteCData(myNorwegianText);
Then I write the file to disk like this:
FileStream myFile = new FileStream(myPath, FileMode.Create);
StreamWriter sw = new StreamWriter(myFile);
stream.Position = 0;
StreamReader sr = new StreamReader(stream);
string content = sr.ReadToEnd();
sw.Write(content);
sw.Flush();
myFile.Flush();
myFile.Close();
Now the problem is that in the file on this, all the Norwegian characters look funny.
I'm probably doing the above in some stupid way. Any suggestions on how to fix it?

Why are you writing the XML first to a MemoryStream and then writing that to the actual file stream? That's pretty inefficient. If you write directly to the FileStream it should work.
If you still want to do the double write, for whatever reason, do one of two things. Either
Make sure that the StreamReader and StreamWriter objects you use all use the same encoding as the one you used with the XmlWriter (not just the StreamWriter, like someone else suggested), or
Don't use StreamReader/StreamWriter. Instead just copy the stream at the byte level using a simple byte[] and Stream.Read/Write. This is going to be, btw, a lot more efficient anyway.

Both your StreamWriter and your StreamReader are using UTF-8, because you're not specifying the encoding. That's why things are getting corrupted.
As tomasr said, using a FileStream to start with would be simpler - but also MemoryStream has the handy "WriteTo" method which lets you copy it to a FileStream very easily.
I hope you've got a using statement in your real code, by the way - you don't want to leave your file handle open if something goes wrong while you're writing to it.
Jon

You need to set the encoding everytime you write a string or read binary data as a string.
Encoding encoding = Encoding.GetEncoding("ISO-8859-1");
FileStream myFile = new FileStream(myPath, FileMode.Create);
StreamWriter sw = new StreamWriter(myFile, encoding);
stream.Position = 0;
StreamReader sr = new StreamReader(stream, encoding);
string content = sr.ReadToEnd();
sw.Write(content);
sw.Flush();
myFile.Flush();
myFile.Close();

As mentioned in above answers, the biggest issue here is the Encoding, which is being defaulted due to being unspecified.
When you do not specify an Encoding for this kind of conversion, the default of UTF-8 is used - which may or may not match your scenario. You are also converting the data needlessly by pushing it into a MemoryStream and then out into a FileStream.
If your original data is not UTF-8, what will happen here is that the first transition into the MemoryStream will attempt to decode using default Encoding of UTF-8 - and corrupt your data as a result. When you then write out to the FileStream, which is also using UTF-8 as encoding by default, you simply persist that corruption into the file.
In order to fix the issue, you likely need to specify Encoding into your Stream objects.
You can actually skip the MemoryStream process entirely, also - which will be faster and more efficient. Your updated code might look something more like:
FileStream fs = new FileStream(myPath, FileMode.Create);
XmlTextWriter xmlTextWriter =
new XmlTextWriter(fs, Encoding.GetEncoding("ISO-8859-1"));
xmlTextWriter.Formatting = Formatting.Indented;
xmlTextWriter.WriteStartDocument(); //Start doc
xmlTextWriter.WriteCData(myNorwegianText);
StreamWriter sw = new StreamWriter(fs);
fs.Position = 0;
StreamReader sr = new StreamReader(fs);
string content = sr.ReadToEnd();
sw.Write(content);
sw.Flush();
fs.Flush();
fs.Close();

Which encoding do you use for displaying the result file? If it is not in ISO-8859-1, it will not display correctly.
Is there a reason to use this specific encoding, instead of for example UTF8?

After investigating, this is that worked best for me:
var doc = new XDocument(new XDeclaration("1.0", "ISO-8859-1", ""));
using (XmlWriter writer = doc.CreateWriter()){
writer.WriteStartDocument();
writer.WriteStartElement("Root");
writer.WriteElementString("Foo", "value");
writer.WriteEndElement();
writer.WriteEndDocument();
}
doc.Save("dte.xml");