Develop Reference

How to extract data from NetCDF file using C# and Microsoft Scientific Data Set?

I am trying to extract data from a NetCDF file using Microsoft's Scientific Data Set libraries (SDS). I'm restricted to using MS SDS and C#. I'm not a programmer by trade so I'm struggling to get the basics working. To begin with I'd like to write a simple script to extract data and write it to a .csv file.
Using the introduction doc and codeplex tutorials. I've tried to write a simple C# console app which just reads the file and writes it out to the console or ideally to a .csv file.
using SDS 1.3 command line I can view the contents of my test file:
D:\NetCDF>sds list test.nc
[2] ACPR of type Single (Time:85) (south_north:213) (west_east:165)
[1] Times of type SByte (Time:85) (DateStrLen:19)
D:\NetCDF>
My script looks like this:
using System;
using System.IO;
using sds = Microsoft.Research.Science.Data;
using Microsoft.Research.Science.Data.Imperative;
namespace NetCDFConsoleApp
{
class Program
{
static void Main(string[] args)
{
/// Gets the path to the NetCDF file to be used as a data source.
var dataset = sds.DataSet.Open("D:\\NetCDF\\test.nc?openMode=readOnly");
SByte[,] times = dataset.GetData<SByte[,]>("Times");
//Int32[,] times = dataset.GetData<Int32[,]>("Times");
//Single[] longitudes = dataset.GetData<Single[]>("west_east");
//var latitudes = dataset.GetData<Single[]>("south_north");
Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR");
for (int iTime = 50; iTime < 60; iTime++)
{
for (int iLongitude = 130; iLongitude < 150; iLongitude++)
{
for (int iLatitude = 130; iLatitude < 140; iLatitude++)
{
// write output data
float thisValue = dataValues[iTime,iLatitude,iLongitude];
Console.WriteLine(iTime);
Console.WriteLine(iLatitude);
Console.WriteLine(iLongitude);
Console.WriteLine(thisValue);
}
}
}
}
}
}
If I comment out the var Times... line it runs. But I'm struggling to get SDS to read Time Dimension. If I use SByte it complains that the variable doesn't exist. If I use Int32 it complains about converting to string.
System.InvalidOperationException was unhandled
HResult=-2146233079
Message=Requested variable does not exist in the data set
Source=ScientificDataSet
StackTrace:
at Microsoft.Research.Science.Data.Imperative.DataSetExtensions.FindVariable(DataSet dataset, Func`2 predicate)
at Microsoft.Research.Science.Data.Imperative.DataSetExtensions.GetData[D](DataSet dataset, String variableName)
at NetCDFConsoleApp.Program.Main(String[] args) in \\gdc-fs01\user$\prm\Visual Studio 2015\projects\NetCDFConsoleApp\Program.cs:line 16
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:
What am I missing?

I think I have solved the puzzle. The problem is that the 2nd variable is empty. I'm not sure if that is by accident or design.
I have found the Date of when data is supposed to start in a meta data field. I've modified my code now so that it retrieves this, and writes it all to the console.
using System;
using System.IO;
using sds = Microsoft.Research.Science.Data;
using Microsoft.Research.Science.Data.Imperative;
namespace NetCDFConsoleApp
{
class Program
{
static void Main(string[] args)
{
/// Gets the path to the NetCDF file to be used as a data source.
var dataset = sds.DataSet.Open("D:\\NetCDF\\test.nc?openMode=readOnly");
string dt = (string)dataset.Metadata["START_DATE"];
Single[,,] dataValues = dataset.GetData<Single[,,]>("ACPR");
for (int iTime = 50; iTime < 60; iTime++)
{
for (int iLongitude = 130; iLongitude < 150; iLongitude++)
{
for (int iLatitude = 130; iLatitude < 140; iLatitude++)
{
// write output data
float thisValue = dataValues[iTime,iLatitude,iLongitude];
Console.WriteLine(dt.ToString() + ',' + iTime.ToString() + ',' + iLatitude.ToString() + ',' + iLongitude.ToString() + ',' + thisValue.ToString());
}
}
}
Console.ReadLine();
}
}
}
I've really struggled with this so I'm sharing this in the hope it will be of use to someone else.
One thing I really found useful was the discussion tab on Codeplex as it has lots of useful code snippets.

Census Geocoder JSON output convert to Xml dataset using JSON.net in C#

I am creating a .Net app in Visual Studio 2012 that queries an address table in my SQL dB and uses the Census Geocoding API to return the specific MSA for each address. I have existing code for the dB query, but I am having trouble with converting the Json output of the Census API to an Xml dataset. I am using Json.net to serialize the json output and then deserialize to .net in order to load into an XmlDocument. Unfortunately, I keep getting an XmlException error:
Data at the root level is invalid. Line 1, position 1
Details:
System.Xml.XmlException was unhandled HResult=-2146232000
Message=Data at the root level is invalid. Line 1, position 1.
Source=System.Xml LineNumber=1 LinePosition=1 SourceUri=""
StackTrace:
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)
at System.Xml.XmlDocument.Load(XmlReader reader)
at System.Xml.XmlDocument.LoadXml(String xml)
at ConsoleApplication1.Program.Main(String[] args) in c:\Users\jdsmith\Documents\Visual Studio
2012\Projects\C#\MSA_Application_v2\MSA_Application_v2\Model\Program.cs:line
54
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext
executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean
preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart() InnerException:
I believe either the Json or the Xml need to be formatted further, but I don't know how. Also, I'm sure I am making this too difficult on myself...if there is a better way, I am all ears.
Here is the sample geolookup I am using to test:
http://geocoding.geo.census.gov/geocoder/geographies/address?street=4600+Silver+Hill+Rd&city=Suitland&state=MD&benchmark=Public_AR_Census2010&vintage=Census2010_Census2010&layers=14&format=json
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using System.Threading.Tasks;
using System.Data;
using System.Net;
using System.IO;
using System.Xml;
using System.Runtime.Serialization.Json;
using System.Xml.Linq;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
namespace ConsoleApplication1
{
class Program
{
private static string geoRT = "geographies";
private static string geoST = "address";
private static string geoStreet = "4600+Silver+Hill+Rd";
private static string geoCity = "Suitland";
private static string geoState = "MD";
private static string geoZip = "20746";
private static string geoBM = "Public_AR_Census2010";
private static string geoVin = "Census2010_Census2010";
private static string geoLayer = "all";
private static string geoFormat = "json";
static void Main(string[] args)
{
StringBuilder geoRelURI = new StringBuilder();
geoRelURI.AppendFormat(#"{0}/{1}?street={2}&city={3}&state={4}&zip={5}&benchmark={6}&vintage={7}&layers={8}&format={9}"
, geoRT, geoST, geoStreet, geoCity, geoState, geoZip, geoBM, geoVin, geoLayer, geoFormat);
Uri geoBaseURI = new Uri("http://geocoding.geo.census.gov/geocoder/");
Uri geoURI = new Uri(geoBaseURI, geoRelURI.ToString());
//Console.WriteLine(geoURI);
//Console.ReadLine();
WebRequest geoRequest = WebRequest.Create(geoURI);
WebResponse geoResponse = geoRequest.GetResponse();
Stream geoDataStream = geoResponse.GetResponseStream();
StreamReader geoReader = new StreamReader(geoDataStream);
string geoString = geoReader.ReadToEnd();
var jsonConvert = JsonConvert.SerializeObject(geoString);
string jsonString = jsonConvert.ToString();
var xmlConvert = JsonConvert.DeserializeObject(jsonString);
string xmlString = xmlConvert.ToString();
XmlDocument geoXMLDoc = new XmlDocument();
geoXMLDoc.LoadXml(xmlString);
XmlWriterSettings xmlSettings = new XmlWriterSettings();
xmlSettings.Indent = true;
XmlWriter geoXMLWriter = XmlWriter.Create("geoXML.xml", xmlSettings);
geoXMLDoc.Save(geoXMLWriter);
Console.Write("<BR>" + geoXMLDoc.OuterXml);
//Console.WriteLine(xmlString);
//Console.ReadLine();
geoDataStream.Close();
geoResponse.Close();
}
}
}

First of all, you are passing a JSON string to geoXMLDoc.LoadXml(). That's not going to work. What you want to do is to convert the JSON to an XmlDocument via JsonConvert.DeserializeXmlNode.
However, some of your JSON properties contain characters that are invalid for use in XML names, in specific whitespace:
{"Census Blocks":[{"BLKGRP":"1",
It seems that this causes DeserializeXmlNode to throw an exception. Thus you'll need to rename the names:
var obj = JObject.Parse(geoString);
foreach (var fix in (from property in obj.Descendants().OfType<JProperty>()
let newName = XmlConvert.EncodeLocalName(property.Name.Replace(" ", ""))
where newName != property.Name
select new { Old = property, New = new JProperty(newName, property.Value) })
.ToList())
{
fix.Old.Replace(fix.New);
}
var xmldoc = JsonConvert.DeserializeXmlNode(obj.ToString());

Need you to post what you are attempting to load into the XmlDocument. That is where you are hitting your problem. If you are trying to load the JSON you get from the web call it won't work, if you are (as I suspect) using JSON.Net to convert the JSON to Xml, the Xml is missing something that the XmlDocument wants. Could be the Xml declaration line, or your xml fragment may not include a root node. Without seeing the xml we have no way to tell specifically what is missing or malformed.

C# UnauthorizedAccessException in File.Copy

I am brushing up on my C# so I decided to write a program that I can use to easily import photos that I take. A little background...I shoot photos in JPEG and RAW and then go through and pick through the JPEGs since they are smaller and easier to handle/preview. I then import only those RAW files that are worth messing with in post production.
I wanted to write a simple program to copy the RAW files from one directory that match the JPEGs that I've sifted through in another.
Here is the code:
static void Main(string[] args)
{
Console.WriteLine("Enter the JPEG Origin Directory: ");
string originDirectory = #"C:\Users\Greg\Pictures\Summer 2013\Back Bay\testJPEG";
Console.WriteLine("Enter the RAW Origin Directory: ");
string copyDirectory = #"C:\Users\Greg\Pictures\Summer 2013\Back Bay\testRAW";
Console.WriteLine("Enter the RAW Import Directory: ");
string rawCopyDirectory = #"C:\Users\Greg\Pictures\Summer 2013\Back Bay\testRAWImport";
char[] delimiterChars = { '_', '.' };
List<string> filesToCopy = new List<string>();
List<string> CopiedFiles = new List<string>();
foreach (var filePath in Directory.GetFiles(originDirectory))
{
Console.WriteLine("Filepath: '{0}'", filePath);
string[] words = filePath.Split(delimiterChars);
filesToCopy.Add(words[1]);
}
filesToCopy.ForEach(Console.WriteLine);
foreach (var copyFilePath in Directory.GetFiles(copyDirectory))
{
string[] delimited = copyFilePath.Split(delimiterChars);
if (filesToCopy.Contains(delimited[1]))
{
Console.WriteLine("Copied: '{0}'", copyFilePath);
string fileName = Path.GetFileName(copyFilePath);
string sourcePath = Path.GetDirectoryName(copyFilePath);
string targetPath = rawCopyDirectory;
string sourceFile = System.IO.Path.Combine(sourcePath, fileName);
string destFile = System.IO.Path.Combine(targetPath, fileName);
System.IO.File.Copy(sourcePath, destFile, true);
}
}
Console.WriteLine("Press any key to exit.");
Console.ReadKey();
}
Everything seems to be working as I'd expect when I write all the variables to the console, however I'm getting an exception on Copy.File that indicates the files are read only. I checked, and they aren't, however the folder itself is, and despite my best efforts I cannot unflag my test folders as readonly. Any help would be appreciated, I've pasted the exception log below.
System.UnauthorizedAccessException was unhandled
HResult=-2147024891
Message=Access to the path 'C:\Users\Greg\Pictures\Summer 2013\Back Bay\testRAW' is denied.
Source=mscorlib
StackTrace:
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at System.IO.File.Copy(String sourceFileName, String destFileName, Boolean overwrite)
at ConsoleApplication1.Program.Main(String[] args) in C:\Users\Greg\documents\visual studio 2010\Projects\Photo Importer\Photo Importer\photoImporter.cs:line 56
at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.AppDomain.nExecuteAssembly(RuntimeAssembly assembly, String[] args)
at System.Runtime.Hosting.ManifestRunner.Run(Boolean checkAptModel)
at System.Runtime.Hosting.ManifestRunner.ExecuteAsAssembly()
at System.Runtime.Hosting.ApplicationActivator.CreateInstance(ActivationContext activationContext, String[] activationCustomData)
at System.Runtime.Hosting.ApplicationActivator.CreateInstance(ActivationContext activationContext)
at System.Activator.CreateInstance(ActivationContext activationContext)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssemblyDebugInZone()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:

The problem can be that you can not delete or overwrite read-only files.
The solution is to change the attributes.
if(File.Exists(destFile))
{
File.SetAttributes(destFile, FileAttributes.Normal);
}
File.Copy(sourcePath, destFile, true);

You are trying to access a file outside of what your program can use.
Try looking at this older post Why is access to the path denied?

Turns out I was calling the wrong variable in File.Copy, and instead was trying to copy a path instead of a file (derp). Everything works now! Thanks for the replies!

subversion authentication with sharpsvn driver

I've set up my subversion server following the steps on : http://www.codinghorror.com/blog/2008/04/setting-up-subversion-on-windows.html
The server is running on my localhost
Configurations
this is my config file:
anon-access = read
auth-access = write
password-db = passwd
authz-db = authz
authz:
[/]
* = w
passwd:
[users]
plorio = pass
test = pass
Code using sharpSvn driver
var svn = new SvnClient();
svnClient.Authentication.DefaultCredentials = new NetworkCredential("test",
"pass",
new Uri("svn://localhost/main").AbsoluteUri);
var localFilename = "C:\\testFile.txt";
var commitArgs = new SvnCommitArgs();
commitArgs.LogMessage = "test";
svn.Commit(localFilename, commitArgs); // <<< error:
Error
svn.Commit(localFilename, commitArgs); gives the following error:
Commit failed (details follow):
innerException : {"Authentication error from server: Username not found"}
SvnErrorCode : SharpSvn.SvnErrorCode.SVN_ERR_RA_NOT_AUTHORIZED
SubversionErrorCode : 170001
StackTrace :
at SharpSvn.SvnClientArgs.HandleResult(SvnClientContext client, SvnException error) in f:\qqn\sharpsvn-dist-1.6\src\sharpsvn\svnclientargs.cpp:line 76
at SharpSvn.SvnClientArgs.HandleResult(SvnClientContext client, svn_error_t* error) in f:\qqn\sharpsvn-dist-1.6\src\sharpsvn\svnclientargs.cpp:line 42
at SharpSvn.SvnClient.Commit(ICollection`1 paths, SvnCommitArgs args, SvnCommitResult& result) in f:\qqn\sharpsvn-dist-1.6\src\sharpsvn\commands\commit.cpp:line 136
at SharpSvn.SvnClient.Commit(String path, SvnCommitArgs args) in f:\qqn\sharpsvn-dist-1.6\src\sharpsvn\commands\commit.cpp:line 74
at Zephyr.OnDemand.WorkbookManagementService.WorkbookManagementOperations.UpdateWorkbook(String client, ManagedWorkbookDetails details, Stream content) in C:\src\zod\ci-tests\SVN\WorkbookManagementService\WorkbookManagementOperations.cs:line 165
at SVN.Program.Main(String[] args) in C:\src\zod\ci-tests\SVN\Program.cs:line 24
at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
I am fairly confident the error exists in the subversion configuration or the svnClient.Authentication.DefaultCredentials if anyone has any ideas it would be helpful. Thanks in advance.

Problem
I was copying over my working directory from another Subversion working directory. This was problematic because I was also copied over the .svn folders. This caused the program to try and commit to the server defined in the .svn folder, which rendered the authentication error.
Solution
Delete .svn folders from working directory and re add and commit files

Can't connect in C# and google spreadsheet api

Trying to access google spreadsheets using their api. following their example, the code doesnt work, and its not obvious why. All I'm trying to do is to connect, and I keep getting back the same error. This is with their code set as of 4/15/10. Can anyone offer any suggestion on what I'm doing wrong?
CodE:
using System;
using Google.GData.Client;
using Google.GData.Extensions;
using Google.GData.Spreadsheets;
namespace google_spreadsheet
{
class Program
{
static void Main(string[] args)
{
SpreadsheetsService myService = new SpreadsheetsService("MySpreadsheet" );
myService.setUserCredentials("account#gmail.com", "xxxxxxx");
string token1 = myService.QueryClientLoginToken();
Console.WriteLine("token is {0}", token1);
Console.ReadLine();
SpreadsheetQuery query = new SpreadsheetQuery();
SpreadsheetFeed feed = myService.Query(query);
Console.WriteLine("list");
foreach (SpreadsheetEntry entry in feed.Entries)
{
Console.WriteLine("Value: {0}", entry.Title.Text);
When I run this, it keeps erroring out at the myService.Query statement, with the following error:
Google.GData.Client.GDataRequestException was unhandled
Message=Execution of request failed: http://spreadsheets.google.com/feeds/spreadsheets/private/full
Source=Google.GData.Client
ResponseString=<HTML>
<HEAD>
<TITLE>Not Found</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Not Found</H1>
<H2>Error 404</H2>
</BODY>
</HTML>
StackTrace:
at Google.GData.Client.GDataRequest.Execute()
at Google.GData.Client.GDataGAuthRequest.Execute(Int32 retryCounter)
at Google.GData.Client.GDataGAuthRequest.Execute()
at Google.GData.Client.Service.Query(Uri queryUri, DateTime ifModifiedSince, String etag, Int64& contentLength)
at Google.GData.Client.Service.Query(Uri queryUri, DateTime ifModifiedSince)
at Google.GData.Client.Service.Query(FeedQuery feedQuery)
at Google.GData.Spreadsheets.SpreadsheetsService.Query(SpreadsheetQuery feedQuery)
at google_spreadsheet.Program.Main(String[] args) in C:\Development Items\VS Projects\VS2008\google_spreadsheet\google_spreadsheet\Program.cs:line 21
at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException: System.Net.WebException
Message=The remote server returned an error: (404) Not Found.
Source=System
StackTrace:
at System.Net.HttpWebRequest.GetResponse()
at Google.GData.Client.GDataRequest.Execute()
InnerException:
Yet, I can take the url
http://spreadsheets.google.com/feeds/spreadsheets/private/full
and manually type it in with my username/password, and it works fine. Any suggestions?
thanks
rocky sanders

I had a similar problem with Mono. In my case it was a problem with certificates, used by SSL.

i believe it's becus u din't specify Uri in document query object
try this instead
DocumentsService service = new DocumentsService("appName");
service.Credentials = new GDataCredentials("email#gmail.com", "password");
DocumentQuery query = new DocumentQuery("http://docs.google.com/feeds/default/private/full/-/contents");
query.Categories.Add(DocumentsListQuery.SPREADSHEETS);
var spreadsheets = service.Query(query).Entries.Cast<DocumentEntry>();