Develop Reference

What is the right way to store queries in a C# program?

I have written a program that uses an Azure API to run queries against the Azure log analytics repository.
The language used to write this query is called Kusto.
I have a lot of Kusto queries that I used directly in my C# code.
Something like this :
Client.ExecuteQuery(#"Heartbeat
| summarize LastCall = max(TimeGenerated) by Computer
| where LastCall < ago(15m)
| take 20");
Some queries are very long and make the code very ugly.
Is there any better and more appropriate way to store these queries say for example in a database that will help to remove them from code.

You can save the queries as text files along with your source code, and then package them with your application as "Embedded Resources".
The accepted answer in this post explains how: How to read embedded resource text file
You can also find plenty of examples and tutorials through your favorite search engine; try "c# embedded resource"

For EF migration I use sql which stored in Data layer library in the resources and after that call like this:
private string ReadSqlFile(string relativePath)
{
var path = Path.Combine(AppContext.BaseDirectory, relativePath);
return File.ReadAllText(path);
}
protected override void Up(MigrationBuilder migrationBuilder)
{
//some migrations methods
var script = ReadSqlFile("Migrations/DataMigration/AddNewCountryFieldToStudent.sql");
migrationBuilder.Sql(script);
}
It's very useful to store in separate files. And don't forget to change property to copy locally.

Fortify cleanse rule doesn't cleanse everything

My team has recently started using Fortify Static Code Analyzer (version 17.10.0 156) on our .NET code base (C# 6 and VB.NET), and are experiencing some pain with the amount of false positives it reports. For any given issue we can't know if it is a false positive without looking at it, and we don't want any actual problems to get lost in the clutter.
We have a utilities library with a method ReadEmbeddedSql which extracts sql from resources embedded in the assembly to execute. Fortify flags any OracleCommand (from Oracle.ManagedDataAccess.Client) which execute the sql returned from this method with a Sql Injection vulnerability.
This vulnerability is reported at the point the sql is set on the command, be it via constructor, or via the CommandText property.
It does not do this if the ReadEmbeddedSql method is defined in the local assembly.
A pared down listing of the source code which produces this result follows below. In the example code, ExecuteSomeSql() and ExecuteSomeSqlDifferently() are flagged with a vulnerability where ExecuteSomeLocalSql() is not.
For Analysis Evidence it only lists the line the OracleCommand is created:
TestDao.cs:27 - OracleCommand()
RuleID: 31D4607A-A3FF-447C-908A-CA2BBE4CE4B7
in the details for the issue it provides:
On line 27 of TestDao.cs, the method ExecuteSomeSql() invokes a SQL query
built using input coming from an untrusted source. This call could
allow an attacker to modify the statement's meaning or to execute
arbitrary SQL commands.
A sample diagram presented by Fortify for this issue:
After much searching, I came across this post describing a similar problem and proposed solution: Can I register a trusted source for SQL statements
After following the instructions there, and verifying the instructions independently in a different version of the user guide (page 90)
The result is unchanged. I added an additional 'SQL Injection Validation Rule' rule which is specifically described as "... identifies a function that properly validates data before using them in a SQL query."
Still no avail.
EDIT:
I have played around with custom rules more, and have been able to determine that the CustomCleanseRules are actually being applied (they do remove other types of taint), but not removing some trust specific flag Fortify applies to our in-house library.
Every value returned from any method of my libraries is distrusted, and none of the rules I've created seem to be able to remove this distrust.
Is there something I am doing wrong, or does Fortify just not work?
Is there a different type of rule needed to cleanse this general distrust?
Example Source code:
In library:
namespace Our.Utilities.Database
{
public abstract class BaseDao
{
protected string ReadEmbeddedSql(string key)
{
//... extract sql from assembly
return sql;
}
}
}
In application:
namespace Our.Application.DataAccess
{
public class TestDao: Our.Utilities.Database.BaseDao
{
public void ExecuteSomeSql()
{
//... connection is created
// Fortify Does not trust sqlText returned from library method.
var sqlText = ReadEmbeddedSql("sql.for.ExecuteSomeSql");
using(var someSqlCommand = new OracleCommand(sqlText, connection)) // Fortify flags creation of OracleCommand as SqlInjection vulnerability.
{
someSqlCommand.ExecuteNonQuery();
}
}
public void ExecuteSomeSqlDifferently()
{
//... connection is created
// Fortify Does not trust sqlText returned from library method.
var sqlText = ReadEmbeddedSql("sql.for.ExecuteSomeSql");
using(var someSqlCommand = connection.CreateCommand())
{
someSqlCommand.CommandText = sqlText; //Fortify flags setting CommandText as SqlInjection vulnerability.
someSqlCommand.ExecuteNonQuery();
}
}
public void ExecuteSomeLocalSql()
{
//... connection is created
var sqlText = ReadEmbeddedSqlLocallyDefined("sql.for.ExecuteSomeSql");
using(var someSqlCommand = new OracleCommand(sqlText, connection))
{
someSqlCommand.ExecuteNonQuery();
}
}
protected string ReadEmbeddedSqlLocallyDefined(string key)
{
//... extract sql from assembly
return sql;
}
}
}
XML of custom rules:
<?xml version="1.0" encoding="UTF-8"?>
<RulePack xmlns="xmlns://www.fortifysoftware.com/schema/rules">
<RulePackID>5A78FC44-4EEB-49C7-91DA-6564805C3F23</RulePackID>
<SKU>SKU-C:\local\path\to\custom\rules\Our-Utilities</SKU>
<Name><![CDATA[C:\local\path\to\custom\rules\Our-Utilities]]></Name>
<Version>1.0</Version>
<Description><![CDATA[]]></Description>
<Rules version="17.10">
<RuleDefinitions>
<DataflowCleanseRule formatVersion="17.10" language="dotnet">
<RuleID>7C49FEDA-AA67-490D-8820-684F3BDD58B7</RuleID>
<FunctionIdentifier>
<NamespaceName>
<Pattern>Our.Utilities.Database</Pattern>
</NamespaceName>
<ClassName>
<Pattern>BaseDao</Pattern>
</ClassName>
<FunctionName>
<Pattern>ReadSqlTemplate</Pattern>
</FunctionName>
<ApplyTo implements="true" overrides="true" extends="true"/>
</FunctionIdentifier>
<OutArguments>return</OutArguments>
</DataflowCleanseRule>
<DataflowCleanseRule formatVersion="17.10" language="dotnet">
<RuleID>14C423ED-5A51-4BA1-BAE1-075E566BE58D</RuleID>
<TaintFlags>+VALIDATED_SQL_INJECTION</TaintFlags>
<FunctionIdentifier>
<NamespaceName>
<Pattern>Our.Utilities.Database</Pattern>
</NamespaceName>
<ClassName>
<Pattern>BaseDao</Pattern>
</ClassName>
<FunctionName>
<Pattern>ReadSqlTemplate</Pattern>
</FunctionName>
<ApplyTo implements="true" overrides="true" extends="true"/>
</FunctionIdentifier>
<OutArguments>return</OutArguments>
</DataflowCleanseRule>
</RuleDefinitions>
</Rules>
</RulePack>

When I run your sample code (I do have to modify it since it will not compile as is). When I run it with SCA 17.10 with 2017Q3 (I also did this on 2017Q2) I did not get a the same SQL Injection Rule ID as you.
Looking at your Analysis Evidence, I assume that the analyzer that found this was not Dataflow or Control flow but maybe the Semantic or Structural?
You can see the type of analyzer that found the finding by looking at the summary tab:
Either way, I don't think a Custom Rule is what I would do here.
An option you can do is to use a Filter file.
This a file that can contain
RuleIDs
InstanceIDs
Categories
When this file is passed into the scan command, any finding that matches any of the fields in the filter file will be filtered out from the results.
You can find examples of using the filter file in <SCA Install Dir>\Samples\Advanced\filter
or you can look in the Fortify SCA Users Guide in Appendix B: Filtering the Analysis
*Note: Your analysis of using filters (in comment) are spot on.

Reading in a complex text file to input into database

I am working on a program that will read in a text file and then insert areas of the text file into different columns on a database. The text file is generally set up like this:
"Intro information"
"more Intro information"
srvrmgr> "information about system"
srbrmgr> list parameters for component *ADMBatchProc*
"Headers"
*Name of record* *alias of record* *value of record*
The columns create a table containing all of the setting information for this component. One all of the settings are listed, the file moves to another component and returns all the information for that component in a new table. I need to read in the component and the information on the tables without the headers or the other information. I will then need to be able to transfer that data into a database. The columns are fixed width on each table within the file.
Any recommendations about how to approach this are welcome. I have never read in a file this complex so I dont really know how to approach ignoring alot of information while trying to get other information ready for a database. Also the component value I am trying to gather always follows the word component on a line that starts with "srvrmgr".
The '*' represents areas that will be put into datbase.
Siebel Enterprise Applications Siebel Server Manager, Version 8.1.1.11 [23030] LANG_INDEPENDENT
Copyright (c) 1994-2012, Oracle. All rights reserved.
The Programs (which include both the software and documentation) contain
proprietary information; they are provided under a license agreement containing
restrictions on use and disclosure and are also protected by copyright, patent,
and other intellectual and industrial property laws. Reverse engineering,
disassembly, or decompilation of the Programs, except to the extent required to
obtain interoperability with other independently created software or as specified
by law, is prohibited.
Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of
Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.
If you have received this software in error, please notify Oracle Corporation
immediately at 1.800.ORACLE1.
Type "help" for list of commands, "help <topic>" for detailed help
Connected to 1 server(s) out of a total of 1 server(s) in the enterprise
srvrmgr> configure list parameters show PA_NAME,PA_ALIAS,PA_VALUE
srvrmgr>
srvrmgr> list parameters for component ADMBatchProc
PA_NAME PA_ALIAS PA_VALUE
---------------------------------------------------------------------- ------------------------------------- --------------------------------------------------------------------------------------------------------------------
ADM Data Type Name ADMDataType
ADM EAI Method Name ADMEAIMethod Upsert
ADM Deployment Filter ADMFilter
213 rows returned.
srvrmgr> list parameters for component ADMObjMgr_enu
PA_NAME PA_ALIAS PA_VALUE
---------------------------------------------------------------------- ------------------------------------- --------------------------------------------------------------------------------------------------------------------
AccessibleEnhanced AccessibleEnhanced False
This is the beginning of the text file. It a produced in a system called Siebel to show all of the settings for this environment. I need to pull the component name (there are multiple on the actual file but the ones shown here are 'ADMBatchProc' and 'ADMObjMgr_enu'), and then the data shown on the table below it that was created by Siebel. The rest of the information is irrelevant for the purpose of the task I need.

I would recommend using Test-Driven Development techniques in this case. I'm guessing that your possible variations of input format are near infinite.
Try this:
1) Create an interface that will represent the data operations or parsing logic you expect the application to perform. For example:
public interface IParserBehaviors {
void StartNextComponent();
void SetTableName(string tableName);
void DefineColumns(IEnumerable<string> columnNames);
void LoadNewDataRow(IEnumerable<object> rowValues);
DataTable ProduceTableForCurrentComponent();
// etc.
}
2) Gather as many small examples of discrete inputs that have well-defined behaviors as possible.
3) Inject a behaviors handler into your parser. For example:
public class Parser {
private const string COMPONENT_MARKER = "srvrmgr";
private readonly IParserBehaviors _behaviors;
public Parser(IParserBehaviors behaviors) {
_behaviors = behaviors;
}
public void ReadFile(string filename) {
// bla bla
foreach (string line in linesOfFile) {
// maintain some state
if (line.StartsWith(COMPONENT_MARKER)) {
DataTable table = _behaviors.ProduceTableForCurrentComponent();
// save table to the database
_behaviors.StartNextComponent();
}
else if (/* condition */) {
// parse some text
_behaviors.LoadNewDataRow(values);
}
}
}
}
4) Create tests around the expected behaviors, using your preferred mocking framework. For example:
public void FileWithTwoComponents_StartsTwoNewComponents() {
string filename = "twocomponents.log";
Mock<IParserBehaviors> mockBehaviors = new Mock<IParserBehaviors>();
Parser parser = new Parser(mockBehaviors.Object);
parser.ReadFile(filename);
mockBehaviors.Verify(mock => mock.StartNextComponent(), Times.Exactly(2));
}
This way, you will be able to integrate under controlled tests. When (not if) someone runs into a problem, you can distill what case wasn't covered, and add a test surrounding that behavior, after extracting the case from the log being used. Separating concerns this way also allows your parsing logic to be independent from your data operation logic. The needs of parsing specific behaviors seems to be central to your application, so it seems like a perfect fit for creating some domain-specific interfaces.

You'll want to read the text file using StreamReader:
using (FileStream fileStream = File.OpenRead(path))
{
byte[] data = new byte[fileStream.Length];
for (int index = 0; index < fileStream.Length; index++)
{
data[index] = (byte)fileStream.ReadByte();
}
Console.WriteLine(Encoding.UTF8.GetString(data)); // Displays: your file - now you can decide how to manipulate it.
}
Perhaps then you'll use Regex to capture the date you'd like to insert:
You might insert into the db like this:
using (TransactionScope transactionScope = new TransactionScope())
{
using (SqlConnection connection = new SqlConnection(connectionString))
{
connection.Open();
SqlCommand command1 = new SqlCommand(
“INSERT INTO People ([FirstName], [LastName], [MiddleInitial])
VALUES(‘John’, ‘Doe’, null)”,
connection);
SqlCommand command2 = new SqlCommand(
“INSERT INTO People ([FirstName], [LastName], [MiddleInitial])
VALUES(‘Jane’, ‘Doe’, null)”,
connection);
command1.ExecuteNonQuery();
command2.ExecuteNonQuery();
}
transactionScope.Complete();
}
Examples adapted from Wouter de Kort's C# 70-483.

What's a good way to store unchanging variables in a C# project?

I'm just learning C#, so pardon my n00bness, please.
We use Visual Studio 2012 to put together a C#/Selenium project for automated testing.
We have three credit card processors.
I want to store the login credentials, test settings, and test payment items for each processor. I could do this in Python in a minute, using a dict:
processorCreds = {
'ProcOne': {
'website': '[url]',
'username': '[username]',
'password': '[password]'
},
'ProcTwo': {
'website': '[url]',
'username': '[username]',
'password': '[password]'
},
}
And then I'd just call it when I need it:
def openProcessor(procName):
urllib2.urlopen(processorCreds[procName]['website'])
openProcessor('ProcTwo')
I want this in C#, basically. I will only rarely need to change these credentials, and they'll be used in multiple test cases, so I want to be able to pull the information wherever I need it.
What would be the best way for me to put this together? Should I use an XML file, or can this be in a class, or...?
[EDIT] The alternative I see is that we set the variables each time we have a test case, which doesn't seem... object-oriented. Or, I suppose I could put the variables on the [Processor]Portal.cs pages... I was just hoping for a way to put alla this in one place with a minimum of fuss, for our occasional "this works for every processor" tests.
(Also this is totally test data, and would be accessible to the same folk who can already see it now, so I'm not worried.)

.NET offers many ways of storing constant or nearly constant data: you have a choice among multiple ways of reading XML files, configuration files, resource files with your own format, and so on. If you would like to define a structure like yours in code, you can use IDictionary:
internal static readonly IDictionary<string,dynamic> ProcessorCreds =
new Dictionary<string,dynamic> {
{"ProcOne", new {
Website = "[url]",
Username = "[username]",
Password = "[password]"
},
{"ProcTwo", new {
Website = "[url]",
Username = "[username]",
Password = "[password]"
}
};
This creates an instance of Dictionary that maps string objects to anonymous objects with three properties - Website, Username, and Password. Since objects that go in belong to an anonymous class (note the lack of class name between new and the opening curly brace) the value type of the dictionary is defined as dynamic. This would let you add more attributes in the future without changing anything else.
You can use ProcessorCreds like this:
urlopen(ProcessorCreds[procName].Website);

You can put them in App.Config( Web.Config) in the appsettings section and use the configurationmanager to get he values as shown in the example below
<appSettings>
<add key="username" value="me" />
<add key="password" value="getmein" />
</appSettings>
In the code you will have the following
string username=ConfigurationManager.AppSettings["username"];

Using C# or Python does not change the answer to this question substantially. If using a mutable in-memory data structure like a dictionary did the trick fine for you in Python, then you can just do the same in C#.
A similar answer uses dynamic, but it is more idiomatic and there are many advantages in C# to favor using static typing, like so:
public class Processor {
public Uri Website { get; private set; }
public string Username { get; private set; }
public string Password { get; private set; }
public Processor(Uri website, string username, string password) {
Website = website;
Username = username;
Password = password;
}
}
var processorCreds = new Dictionary<string, Processor> {
{ "ProcOne", new Processor(new Uri("[url]"), "[username]", "[password]") },
{ "ProcTwo", new Processor {new Uri("[url]"), "[username]", "[password]") }
};
which case be used as
processorCreds["ProcOne"].Website

There are a lot of different ways to do this. You could store the data in a static readonly dictionary.
public static readonly Dictionary<int, int> MY_DICTIONARY;
And place this inside a static class available throughout your project.
You could also store data in the Properties settings.settings file.
Generally, in my C# code I try to avoid global data and instead pass information when it is needed. This allows for code that is more easily tested.

Your question has more security and procedural implications than technical.
Since you're saving usernames and passwords for payment processing, there're all kinds of risks to storing these where they are publicly available. Do you need PCI compliance in your business? Given that this is security information, don't you want to keep them under lock-and-key to some extent? Perhaps keep them in a database, or encrypted in some way.
If there's no such issue, your appConfig file is probably the best bet: you can define them in your project, and retrieve them using the ConfigurationManager class. The will be stored in the XML file that is deployed along with your application. In plain text.

Put them in a static dynamic variable - this is the most pythonish way to do it, although it's not idiomatic c#
static dynamic processorCreds =new {
ProcOne= new {
website= "[url]",
username= "[username]",
password= "[password]"
},
ProcTwo= new {
website= "[url]",
username= "[username]",
password= "[password]"
},
}
You can then access it using var s = (string) processorCreds.ProcTwo.website;

c# to AS400 / JD Edwards - run program - Data source name not found and no default driver specified

UPDATE: The code example below works. I changed the connection string. That was the problem. For an alternative (non-ADO) solution, see Mike Wills's link in the comments.
I have a c# class that (if I ever get it working) runs a program written in RPG code on AS/400 - JD Edwards. I know next to nothing about AS/400 and/or JD Edwards.
I've got other classes in my (intranet) web app that connect to JD Edwards, running SQL queries and setting/getting data. All of these are using the IBM.Data.DB2.iSeries dll and are working great.
For this, I wrote a similar class using the aforementioned dll, but it wasn't working. I even read somewhere online that you can't run a program using this dll. I found that a little fishy, but on suggestion from my JD Edwards co-worker, I scrapped the class and re-wrote it using the adodb dll. No data need be returned from this program run. I just want the program to run.
Here is a dummified version of the class:
private void runJDEProgram() {
ADODB.Connection cn = new ADODB.Connection();
cn.ConnectionString = "Provider=ABABAB;Data Source=111.111.111";
ADODB.Command cmdDetail = new ADODB.Command();
cn.Open(); //has to be open before setting an active connection.
cmdDetail.ActiveConnection=cn;
cmdDetail.CommandType = ADODB.CommandTypeEnum.adCmdText;
cmdDetail.CommandText = "{{CALL BLAH.BLAH(?,?)}}";
cmdDetail.Prepared = true;
cmdDetail.Parameters.Append(cmdDetail.CreateParameter("P1", ADODB.DataTypeEnum.adChar, ADODB.ParameterDirectionEnum.adParamInput, 10, "BLAH123"));
cmdDetail.Parameters.Append(cmdDetail.CreateParameter("P2", ADODB.DataTypeEnum.adChar, ADODB.ParameterDirectionEnum.adParamInput, 10, "BLAH456"));
object dummy = Type.Missing; //otherwise, we couldn't get past this.
cmdDetail.Execute(out dummy, ref dummy, 0);
cn.Close();
}
Here's the error I get when it runs:
{"[Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified"}
Where am I screwing up? Thanks!
EDIT: The connection string works when querying the AS/400 to get/set data. Does it need to be modified for an operation like this, or for use with ADO?

The connection string has to be different for this, for some reason. I don't use the same connection string for the other classes that just run queries. The above connection string worked fine (with the appropriate values, of course).
It will prompt for password, but the JDE folks here wanted that.