I use LinqPad with the MySQL IQ driver to query data from a Magento database. I not only use this for reporting on the database but for doing updates. I can use the standard SubmitChanges() method to update data, but that often ends up with unbearably slow updates that can literally take hours - one of my tables has 35,707 records that I recreate on a regular basis.
So instead I generate SQL statements in my LinqPad queries and then execute them in a separate tab after selecting "SQL" in the language drop-down.
For example, my output might be something like this:
UPDATE catalog_category_product SET position = 6040 WHERE (category_id = 156 AND product_id = 12648);
UPDATE catalog_product_entity_media_gallery_value SET label = 'Sandy Beach' WHERE ((store_id = 0) AND (value_id = 8791));
-- Done.
I have recently found that LinqPad has a nice class called Hyperlinq that allows me to write code like this:
(new Hyperlinq(QueryLanguage.SQL, myGeneratedSqlText, "Run Query")).Dump();
The result is that a hyperlinq is put in the output window that will run the query (in my example the contents of myGeneratedSqlText) in a new tab and execute the query.
This is very convenient.
However, I now want to be able to save a log of queries that are executed. There doesn't seem to be (an easy) built-in way to manually execute a "generated" query in LinqPad. I can certainly use Util.Run to execute an existing saved query, in fact I do something like this:
Util
.OnDemand("Run Query", () =>
{
var fn = createOutputQueryFileName(); // Timestamped query name
System.IO.File.WriteAllText(fn, myGeneratedSqlText);
var run = Util.Run(fn, QueryResultFormat.Text);
var result = run.AsString();
return result.StartsWith("[]") ? "Success" : result;
})
.Dump();
The only drama with this is that I have to prefix the text in myGeneratedSqlText with the following:
var preamble = #"<Query Kind=""SQL"">
<Connection>
<ID>ec026b74-8d58-4214-b603-6d3145e03d7e</ID>
<Driver Assembly=""IQDriver"" PublicKeyToken=""5b59726538a49684"">IQDriver.IQDriver</Driver>
<Provider>Devart.Data.MySql</Provider>
<CustomCxString>[DELETED]</CustomCxString>
<Server>127.0.0.1</Server>
<Database>prod_1_8</Database>
<Password>[DELETED]</Password>
<UserName>[DELETED]</UserName>
<NoPluralization>true</NoPluralization>
<NoCapitalization>true</NoCapitalization>
<DisplayName>Production Tunnel</DisplayName>
<EncryptCustomCxString>true</EncryptCustomCxString>
<Persist>true</Persist>
<DriverData>
<StripUnderscores>false</StripUnderscores>
<QuietenAllCaps>false</QuietenAllCaps>
<Port>6606</Port>
</DriverData>
</Connection>
</Query>
";
I would really like to avoid all of this preamble stuff and include a line like this in my Util.OnDemand(...) code:
var run = Util.Run(QueryLanguage.SQL, myGeneratedSqlText, QueryResultFormat.Text);
(But this method doesn't exist.)
The key requirement here is to display a hyperlinq in the LinqPad output window that, if clicked, will save the query to disk as a log and also execute the query.
Can anyone suggest a clean way for me to do it?
I hope I've understood you correctly. When you've selected a connection in the top bar, your UserQuery becomes a datacontext. For this reason, you can use ExecuteQuery and ExecuteCommand on this within an action based Hyperlinq.
new Hyperlinq(() => {
"do log work here".Dump();
this.ExecuteQuery<string>(generatedSelect).Dump("Results");
this.ExecuteCommand(generatedCommand).Dump("Results");
}, "Run Query").Dump();
Unfortunately this outputs to the current tab, but hopefully this will at least get you most of the way to done :)
Here's an image of it at work:
As you're using MySQL, you can go via the connection property on this:
new Hyperlinq(() => {
"do log work here".Dump();
using (var command = this.Connection.CreateCommand())
{
// Usual command logic here
}
}, "Run Query").Dump();
Related
I have an aggregation query that I run in a mongo shell. What I want, is to execute this command from a simple console application with .Net 5.
I just want to run the command (not use any fancy LINQ queries) and loop in the results to measure somethings.
I tried a simple query:
_client = new MongoClient("mongodb://localhost:27017");
_database = _client.GetDatabase("zzz");
var res = await _database.RunCommandAsync<BsonDocument>("users.find({})");
But results is always null. I haven't quite understood whether runCommand supports only built-in commands or it's more generic.
Could you please elaborate?
Disclaimer: I don't know much about MongoDb.
From the docs:
db.runCommand(
{
"find": <string>,
// ...
}
)
So it seems to me your query should be:
var res = await _database.RunCommandAsync<BsonDocument>("{ find: 'users' }");
A question I can't figure out, I am having fun with a little tool that I am making, the idea is that I allow the user that he/she writes his/her own C# code, and I save that code into a DB, once they wish to run that script I simply execute the following code:
var Script = CSharpScript.Create<string>(ScriptText, options, typeof(Script_Host));
var _Compilation = Script.Compile();
ScriptRunner<string> runner = Script.CreateDelegate();
Script_Host globals = new Script_Host();
globals._Parent = this;
globals._ConnectionString = _ConnectionString;
globals.Source = Source;
globals.Destination = Destination;
globals.Parameters = Parameters;
globals.Result = "";
_ScriptResult = runner(globals).Result;
Result = _ScriptResult;
So I have compiled the code and it's runnable and it works, and that is great, but it has a flaw. It means every time a user wants to run the code I have to take that piece of code, compile it and run it ... this takes time.
Now I simply want to take that compiled code, serialize it and insert it into a DB as varbinary ... so each script gets compiled only once (or more if they are doing an update) and that's it..
What is best way to reach for the compiled script, how can I convert it to varbinary?
You're using the wrong tool for the job.
Scripting is about "quickly executing a string of code" if you want to dynamically make assemblies and keep them around a standard compilation process is the better way to go ...
Executing a simple script:
await CSharpScript.EvaluateAsync("Console.WriteLine(\"Hello world!\")");
Creating assemblies:
https://learn.microsoft.com/en-us/dotnet/framework/reflection-and-codedom/generating-and-compiling-source-code-from-a-codedom-graph
... the result here is that you end up with a physical assembly you can reuse not a context based on the fly generated blob of script in ram.
again i am trying to do something and not sure if it is possible. i want to run my Nunit tests and after each test is run, i want to output the result to my ui. the ui is custom and is used by my test team to run and see test results. when i say it is used, it is not developed yet (fully) :) - bringing me to this question.
my code so far
TestPackage package = new TestPackage(path);
RemoteTestRunner remote = new RemoteTestRunner();
remote.Load(package);
TestResult result = remote.Run(new NullListener(), TestFilter.Empty, true, LoggingThreshold.All);
while (remote.Running)
{
// want to capture results here
if (result.HasResults)
// i can never get here while test is running
}
How about dumping the results as XML into shared location. And then your UI can parse/pick up from that location
Also try to use common XML schema so that you can easily serialize/deserialize back from XML to C# object and vice versa
We have done smth similar in the past and above scenario has worked quite well
I've searched all over and I now have to ask SO. I'm trying to construct a simple dataflow using EzAPI. It's been anything but easy, but I'm committed to figuring this out. What I can't figure out is how to get the EzOleDBDestination working. Here's my complete code
var a = new Application();
// using a template since it's impossible to set up an ADO.NET connection to MySQL
// using EzAPI and potentially even with the raw SSIS API...
var pkg = new EzPackage(a.LoadPackage(#"C:\...\Package.dtsx", null));
pkg.Name = "Star";
var df = new EzDataFlow(pkg);
df.Name = "My DataFlow";
var src = new EzAdoNetSource(df);
src.Name = "Source Database";
src.SqlCommand = "SELECT * FROM enum_institution";
src.AccessMode = AccessMode.AM_SQLCOMMAND;
src.Connection = new EzConnectionManager(pkg, pkg.Connections["SourceDB"]);
src.ReinitializeMetaData();
var derived = new EzDerivedColumn(df);
derived.AttachTo(src);
derived.Name = "Prepare Dimension Attributes";
derived.LinkAllInputsToOutputs();
derived.Expression["SourceNumber"] = "id";
derived.Expression["Name"] = "(DT_STR,255,1252)description";
// EDIT: reordered the operation here and I no longer get an error, but
// I'm not getting any mappings or any input columns when I open the package in the designer
var dest = new EzOleDbDestination(df);
dest.AttachTo(derived, 0, 0);
dest.Name = "Target Database";
dest.AccessMode = 0;
dest.Table = "[dbo].[DimInstitution]";
dest.Connection = new EzConnectionManager(pkg, pkg.Connections["TargetDB"]);
// this comes from Yahia's link
var destInput = dest.Meta.InputCollection[0];
var destVirInput = destInput.GetVirtualInput();
var destInputCols = destInput.InputColumnCollection;
var destExtCols = destInput.ExternalMetadataColumnCollection;
var sourceColumns = derived.Meta.OutputCollection[0].OutputColumnCollection;
foreach(IDTSOutputColumn100 outputCol in sourceColumns) {
// Now getting COM Exception here...
var extCol = destExtCols[outputCol.Name];
if(extCol != null) {
// Create an input column from an output col of previous component.
destVirInput.SetUsageType(outputCol.ID, DTSUsageType.UT_READONLY);
var inputCol = destInputCols.GetInputColumnByLineageID(outputCol.ID);
if(inputCol != null) {
// map the input column with an external metadata column
dest.Comp.MapInputColumn(destInput.ID, inputCol.ID, extCol.ID);
}
}
}
Basically, anything that involves calls to ReinitializeMetadata() results in 0xC0090001, because that method is where the error happens. There's no real documentation to help me, so I have to rely on any gurus here.
I should mention that the source DB is MySQL and the target DB is SQL Server. Building packages like this using the SSIS designer works fine, so I know it's possible.
Feel free to tell me if I'm doing anything else wrong.
EDIT: here's a link to the base package I'm using as a template: http://www.filedropper.com/package_1 . I've redacted the connection details, but any MySQL and SQL Server database will do. The package will read from MySQL (using the MySQL ADO.NET Connector) and write to SQL Server.
The database schema is mostly irrelevant. For testing, just make a table in MySQL that has two columns: id (int) and description (varchar), with id being the primary key. Make equivalent columns in SQL Server. The goal here is simply to copy from one to the other. It may end up being more complex at some point, but I have to get past this hurdle first.
I can't test this now BUT I am rather sure that the following will help you get it working:
Calling ReinitializeMetadata() causes the component to fetch the table metadata. This should only be called after setting the AccessMode and related property. You are calling it before setting AccessMode...
Various samples including advice on debugging problems
define the derived column(s) directly in the SQL command instead of using a EzDerivedColumn
try to get it working with 2 SQL Server DBs first, some of the available MySQL ADO.NET provider have some shortcomings under some circumstances
UPDATE - as per comments some more information on debugging this and a link to a complete end-to-end sample with source:
http://blogs.msdn.com/b/mattm/archive/2009/08/03/looking-up-ssis-hresult-comexception-errorcode.aspx
http://blogs.msdn.com/b/mattm/archive/2009/08/03/debugging-a-comexception-during-package-generation.aspx
Complete working sample with source
I've had this exact same issue and been able to resolve it with a lot of experimentation. In short you must set the connection for both the source and destination, and then call the attachTo after both connections are set. You must call attachTo for every component.
I've written a blog about starting with an SSIS package as a template, and then manipulating it programmatically to produce a set of new packages.
The article explains the issue more.
Why do I need to call the ToList() method on my LINQ query?
For example:
private void btnEnc_Click(object sender, RoutedEventArgs e)
{
SHA1 sha = new SHA1Managed();
string sResult = "";
var v = sha.ComputeHash(
UTF8Encoding.Unicode.GetBytes(tbxWordToEncrypt.Text)
).Select(
p => sResult += string.Format("{0:x2}", p)
).ToList();
Clipboard.SetText(sResult);
tbxEncrypted.Text = sResult;
}
Also, when I try to access the clipboard I get a security dialog box. How can I prevent this?
By default you run under partial trust. When calling ClipBoard.SetText() (or ClipBoard.SetText(...)) the user must confirm access.
If you create an out-of-browser application and request elevated trust, this restriction no longer applies and no dialog box is shown.
You can configure your application to require elevated trust. You need to set this in the application's manifest.
For more information take a look at MSDN:
http://msdn.microsoft.com/en-us/library/ee721083(v=vs.95).aspx
Open your project's properties and navigate to the Silverlight tab.
Check the option "Enable running application out of browser".
Click on the button Out-Of-Browser Settings. A new dialog will popup.
Check the option "Require elevated trust when running outside the browser".
When a user installs your Silverlight application they will get a security warning before they can proceed. This only happens once. When running your application this way the ClipBoard.SetText() call will no longer trigger a security dialog.
The reason that you need to call to list is because the expression inside the Select isn't evaluated until the expression created by the LINQ statement is evaluated. Because you're using it to append to sResult, that variable won't have had its value changed before you put it on the clipboard unless you "run" the LINQ expression using ToList(). Note that the output of ToList() is basically worthless.
The bigger problem is that you're misusing the Select. You really should be using string.Join instead of building the string inside the Select clause. Building it inside the Select clause is going to be unexpected for people reading your code and harder to understand.
var sResult = string.Join( "",
sha.ComputeHash(
UTF8Encoding.Unicode.GetBytes(tbxWordToEncrypt.Text)
).Select(
p => string.Format("{0:x2}", p)
));
As for Linq:
You only need to call ToList() if you want the data to be evaluated imediatly.
Most Linq operators are lazy by design, and it's a good thing.