Well, as the title says. I'd like to use a script component destination, and then utilize LINQ to select which rows to process for output.
For a bit more background, I have this ugly merged thing with a one-to-many relationship. The rows look sort of like:
[ID] [Title] [OneToManyDataID]
1 Item one 2
1 Item one 4
1 Item one 3
3 Item two 1
3 Item two 5
We'll call the objects [Item], which has the ID and Title columns and [OneToMany]
I was hoping I could throw the entire thing to a script component destination, and then use LINQ to do something like group by the item and only take the data from the highest OneToMany object. Sort of like:
foreach(var item in Data.GroupBy(d=>d.Item).Select(d=> new {Item = d.Key})){
//Then pick out the highest OneToMany ID for that row to use with it.
}
I realize there are probably better LINQ queries to accomplish this, but the point is, the script component in SSIS seems to only allow working with it on a per-row basis, with the predefined ProcessInputRow-method. Where I'd like to determine exactly which rows are processed and what properties are passed to that method.
How would I go about doing this?
To restate your problem, how can I make an Script Transformation stop processing row-by-row? By default, a script transformation is going to be a synchronous component - 1 row in, 1 row out. You'll want to change that to an asynchronous component 1 row in - 0 to many rows out.
On your Script Transformation Editor, the Inputs and Outputs tab, for your output collection Output 0 change the value of SynchronousInputID from whatever it is to None.
Don't cast stones on my LINQ code-I trust you can handle making that work right. The intention of this code block is to demonstrate how you would collect your rows for processing and then pass them on to a downstream consumer after modifying them. I commented on the methods to help you understand what each one of them does in the script component life cycle but if you'd rather read MSDN they know a bit more than I do ;)
using System;
using System.Data;
using System.Linq;
using System.Collections.Generic;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
/// <summary>
/// Our LINQ-able thing.
/// </summary>
List<Data> data;
/// <summary>
/// Do our preexecute tasks, in particular, we will instantiate
/// our collection.
/// </summary>
public override void PreExecute()
{
base.PreExecute();
this.data = new List<Data>();
}
/// <summary>
/// This method is called once the last row has hit.
/// Since we will can only find the highest OneToManyDataId
/// after receiving all the rows, this the only time we can
/// send rows to the output buffer.
/// </summary>
public override void FinishOutputs()
{
base.FinishOutputs();
CreateNewOutputRows();
}
/// <summary>
/// Accumulate all the input rows into an internal LINQ-able
/// collection
/// </summary>
/// <param name="Row">The buffer holding the current row</param>
public override void Input0_ProcessInputRow(Input0Buffer Row)
{
// there is probably a more graceful mechanism of spinning
// up this struct.
// You must also worry about fields that have null types.
Data d = new Data();
d.ID = Row.ID;
d.Title = Row.Title;
d.OneToManyId = Row.OneToManyDataID;
this.data.Add(d);
}
/// <summary>
/// This is the process to generate new rows. As we only want to
/// generate rows once all the rows have arrived, only call this
/// at the point our internal collection has accumulated all the
/// input rows.
/// </summary>
public override void CreateNewOutputRows()
{
foreach (var item in this.data.GroupBy(d => d.ID).Select(d => new { Item = d.Key }))
{
//Then pick out the highest OneToMany ID for that row to use with it.
// Magic happens
// I don't "get" LINQ so I can't implement the poster's action
int id = 0;
int maxOneToManyID = 2;
string title = string.Empty;
id = item.Item;
Output0Buffer.AddRow();
Output0Buffer.ID = id;
Output0Buffer.OneToManyDataID = maxOneToManyID;
Output0Buffer.Title = title;
}
}
}
/// <summary>
/// I think this works well enough to demo
/// </summary>
public struct Data
{
public int ID { get; set; }
public string Title { get; set; }
public int OneToManyId { get; set; }
}
Configuration of the Script Transformation
Related
I have a simple aspx page "Default" with a code behind. The code behind has a set of simple methods.
What I'm looking to do is to create a method OnGridViewChange() - see the mockup below - which is called whenever the user changes a row in the gridview. The row updates a local list of Rows and then converts to object to a json string which is showed to the user.
Essentially what I'm building is a glorified json viewer in asp.net web forms which takes a json string, displays it as a table and then updates the json and table on user changes.
I think I have most of the major methods outlined below, but what I need help with is the OnUpdate method,
public partial class _Default : Page
{
public class FoobarRow
{
public string Name { get; set; }
public int NumberOfHats { get; set; }
}
public List<FoobarRow> FoobarRows { get; set; }= new List<FoobarRow>();
/// <summary>
/// Vera important json formatted string. Gets updated when gridview changes
/// </summary>
public string Json { get; set; }
/// <summary>
/// Updates gridview on the page from the json entered by the user on a previous page
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
protected void Page_Load(object sender, EventArgs e)
{
FoobarRows = JsonConvert.DeserializeObject<List<FoobarRow>>(GetJsonFromUserWhenPageLoads());
UpdateGridviewFromList();
}
/// <summary>
/// Actually updates the gridview from the list of FoobarRows
/// </summary>
private void UpdateGridviewFromList()
{
ApplicationGridView.DataSource = FoobarRows;
ApplicationGridView.DataBind();
}
/// <summary>
/// Is called whenever user edits any row in the gridview and saves the changes. Updates the json and the local list of rows
/// </summary>
/// <param name="rows"></param>
public void OnGridViewChange(List<FoobarRow> rows)
{
FoobarRows = rows;
DisplayCurrentJsonOnPageForUser(JsonConvert.SerializeObject(rows));
UpdateGridviewFromList();
}
public string GetJsonFromUserWhenPageLoads()
{
//Finish later
return "some json the user will have entered on a previous page and will be loaded into a label or something of the like";
}
public void DisplayCurrentJsonOnPageForUser(string json)
{
//Finish later
}
}
Having looked around, I found this documentation from Microsoft https://msdn.microsoft.com/en-us/library/ms972948.aspx?f=255&MSPPError=-2147217396 about the aspx gridview but it seems needlessly complicated for my usecase.
Does anyone know of any datagrid plugin for webforms or other asp.net element that would allow me to tie changes to a simple event such as OnGridViewChange(). I'm new to web forms and the above mentioned article from Microsoft is the best I've managed to find but I feel like there must be a better way!
Apparently this can be achieved by using this event model found here.
https://msdn.microsoft.com/en-us/library/system.web.ui.webcontrols.gridview.rowupdating%28v=vs.110%29.aspx?f=255&MSPPError=-2147217396
***Just for learning purpose***
Recently I just knew the word cache and cache mechanism and generally understand that the cache mechanism is a good thing on system responding performance and reduce many interacting with database.
And based on the talking with someone else, they told me the general idea that we can create an independent library and cache the data retrieving from database and once we need it in our business layer, then we can retrieve it from the cache layer.
And they also shared something but not very detailed that the database can update the cache layer automatically when the data in database refreshed, like updating, adding and deleting.
So my questions comes, how does database know and update cache layer proactively and automatically? Can anybody share something with me? or are there any existing frameworks, open source solutions?
I would much appreciate for your kindly help. I'm looking forward to hearing from you my friend.
Try this third party cache: CacheCrow, it is a simple LFU based cache.
Install using powershell command in visual studio: Install-Package CacheCrow
Code Snippet:
// initialization of singleton class
ICacheCrow<string, string> cache = CacheCrow<string, string>.Initialize(1000);
// adding value to cache
cache.Add("#12","Jack");
// searching value in cache
var flag = cache.LookUp("#12");
if(flag)
{
Console.WriteLine("Found");
}
// removing value
var value = cache.Remove("#12");
For more information you can visit: https://github.com/RishabKumar/CacheCrow
Jacob,
Let me give you an example...
In the data layer when we are going to retrieve a list of objects that should be cached from the database we could to something like this.
if (!CacheHelper.Get("AllRoles", out entities))
{
var items = _context.Set<Roles>().ToList();
entities = items;
var cachableEntities = entities.ToList();
CacheHelper.Add(cachableEntities, "AllRoles");
}
return entities;
You'll notice that I have Cache helper that will search the cache for the key "AllRoles" if it finds the cache it will return the entities from the cache. If it cant find it it will get the data from the database and Create the cache with the key.
Additionally, every time we add/delete/or change an item in this table we could simple destroy this cache.
CacheHelper.Clear(CacheKey);
So answering the question, in this sample the database doesn't know when to recreate the cache, the application logic does.
Here a sample of a Cache Helpers you may use....
using System;
using System.Collections.Generic;
using System.Web;
namespace Core.Helpers
{
public static class CacheHelper
{
public static List<string> GetCacheKeys()
{
List<string> keys = new List<string>();
// retrieve application Cache enumerator
var enumerator = System.Web.HttpRuntime.Cache.GetEnumerator();
while (enumerator.MoveNext())
{
keys.Add(enumerator.Key.ToString());
}
return keys;
}
/// <summary>
/// Insert value into the cache using
/// appropriate name/value pairs
/// </summary>
/// <typeparam name="T">Type of cached item</typeparam>
/// <param name="o">Item to be cached</param>
/// <param name="key">Name of item</param>
public static void Add<T>(T o, string key)
{
// NOTE: Apply expiration parameters as you see fit.
// I typically pull from configuration file.
// In this example, I want an absolute
// timeout so changes will always be reflected
// at that time. Hence, the NoSlidingExpiration.
if (HttpContext.Current != null)
HttpContext.Current.Cache.Insert(
key,
o,
null,
DateTime.Now.AddMinutes(1440),
System.Web.Caching.Cache.NoSlidingExpiration);
}
/// <summary>
/// Remove item from cache
/// </summary>
/// <param name="key">Name of cached item</param>
public static void Clear(string key)
{
if (HttpContext.Current != null)
HttpContext.Current.Cache.Remove(key);
}
/// <summary>
/// Check for item in cache
/// </summary>
/// <param name="key">Name of cached item</param>
/// <returns></returns>
public static bool Exists(string key)
{
var exists= HttpContext.Current != null && HttpContext.Current.Cache[key] != null;
return exists;
}
/// <summary>
/// Retrieve cached item
/// </summary>
/// <typeparam name="T">Type of cached item</typeparam>
/// <param name="key">Name of cached item</param>
/// <param name="value">Cached value. Default(T) if
/// item doesn't exist.</param>
/// <returns>Cached item as type</returns>
public static bool Get<T>(string key, out T value)
{
try
{
if (!Exists(key))
{
value = default(T);
return false;
}
value = (T)HttpContext.Current.Cache[key];
}
catch
{
value = default(T);
return false;
}
return true;
}
}
}
I am creating an SQLite Database in Visual Studio with Xamarin in C#.
I should note that this is for android only.
I am trying to make it so I am able to insert data into the SQLite database but I am unsure how.
I have been following this but I'm still unsure.
Here is the method I am trying to create.
/// <summary>
/// Insert a single ping group into the SQLite ping database.
/// </summary>
/// <param name="pingGroup"></param>
public void AddUnsynchronizedPing(PingGroup pingGroup)
{
// TODO: Add the passed ping group parameter into the SQLite database as new/unsynchronized.
if (pingGroup != null)
{
// Add ping group to the database.
// Add pings to the database.
// Maybe one step, maybe done separately.
// If done separately, must set Ping.PingGroupID to ID of original ping group.
}
}
For context, here is the entire class.
namespace BB.Mobile
{
/// <summary>
/// A class to provide a single interface for interacting with all SQLite data operations for stored tracking points.
/// </summary>
///
class DataManager
{
private SQLiteConnection db = null;
public DataManager()
{
if (this.db == null)
{
string dbPath = Path.Combine(
System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal),
"bb.db3");
db = new SQLiteConnection(dbPath);
db.CreateTable<Ping>();
db.CreateTable<PingGroup>();
}
}
/// <summary>
/// Will compile and return all matching unsynchronized ping data from the SQLite database.
/// </summary>
/// <returns></returns>
public List<PingGroup> GetUnsynchronizedPings()
{
List<PingGroup> unsynchronizedPings = new List<PingGroup>();
// TODO: Retrieve all unsynchronized pings from the SQLite database and return them to the caller.
//var pGroup = db.Get<PingGroup>();
//var pGroupList = db.List<PingGroup>();
var pGroups = db.Table<PingGroup>();
foreach (var pGroup in pGroups)
{
}
return unsynchronizedPings;
}
/// <summary>
/// Insert a single ping group into the SQLite ping database.
/// </summary>
/// <param name="pingGroup"></param>
public void AddUnsynchronizedPing(PingGroup pingGroup)
{
// TODO: Add the passed ping group parameter into the SQLite database as new/unsynchronized.
if (pingGroup != null)
{
// Add ping group to the database.
// Add pings to the database.
// Maybe one step, maybe done separately.
// If done separately, must set Ping.PingGroupID to ID of original ping group.
}
}
/// <summary>
/// Mark all open and unsynchronized pings in the database as synchronized.
/// </summary>
public void SetAllPingsSynchronized()
{
db.DeleteAll<PingGroup>();
db.DeleteAll<Ping>();
}
}
}
Thank you in advance.
To insert the object to sqlite database, you can just use something like:
void InsertPing(Ping p)
{
db.Insert(p);
}
void InsertGroupOfPings(IEnumerable<Ping> pings)
{
db.InsertAll(pings);
}
and to retrieve objects (for example):
List<Ping> GetPings()
{
// I assume here that Ping object has property named Synchronized
return db.Query<Ping>("select * from Ping where Synchronized = 0");
}
The SQLite library creates its tables according to your class definitions, so you can think about the properties of the class as of columns inside the table.
In my mvc application during certain times of the year we want to show one of two links. Basically I have to switch the link when I get a call from management. So, I thought instead of having to recompile the app I would add a custom app setting to the web.config file. Then I created a wrapper so that it is strongly typed. Now, my problem is I don't know where to execute the logic. Should add a property to my view model and set it in the controller based on the configuration setting value? Or should I read it directly in my View and toggle between the two links? I'm pretty sure this only belongs in the view or the controller, and not the service layer, since it is used specifically for UI stuff.
Details.cshtml //current code
#if(Search.App.ParcelDetailDisplayMode == Search.App.DisplayMode.Tax ){
<a id="tax-link" href="#taxlink" title="View Tax Bill on Tax Collectors Website">Tax Bill</a>
}
else if(Search.App.ParcelDetailDisplayMode == Search.App.DisplayMode.Trim ){
<a id="trim-link" href="#trimlink" title="View your TRIM notice online">Trim Notice</a>
}
web.config
<add key="ParcelDetailDisplayMode" value="Tax"/>
config wrapper
namespace Search
{
/// <summary>
/// The app.
/// </summary>
public static class App
{
/// <summary>
/// Gets the tax bill link.
/// </summary>
public static string TaxBillLink
{
get
{
return ConfigurationManager.AppSettings["TaxBillLink"];
}
}
/// <summary>
/// Gets the trim notice link.
/// </summary>
public static string TrimNoticeLink
{
get
{
return ConfigurationManager.AppSettings["TrimLink"];
}
}
/// <summary>
/// Gets the map link.
/// </summary>
public static string MapLink
{
get
{
return ConfigurationManager.AppSettings["MapLink"];
}
}
/// <summary>
/// Gets the update address link.
/// </summary>
public static string UpdateAddressLink
{
get
{
return ConfigurationManager.AppSettings["UpdateAddressLink"];
}
}
/// <summary>
/// Gets the release name.
/// </summary>
public static string ReleaseName
{
get
{
return ConfigurationManager.AppSettings["ReleaseName"];
}
}
/// <summary>
/// Gets the parcel detail display mode.
/// </summary>
public static DisplayMode ParcelDetailDisplayMode
{
get
{
var r = DisplayMode.Tax;
DisplayMode.TryParse(ConfigurationManager.AppSettings["ParcelDetailDisplayMode"], out r);
return r;
}
}
/// <summary>
/// The display mode.
/// </summary>
public enum DisplayMode
{
/// <summary>
/// The trim.
/// </summary>
Trim,
/// <summary>
/// The tax.
/// </summary>
Tax
}
}
}
I would say it does not really matter. Adding it as a property of your model feels to give a little bit more separation.
What does matter though is that your wrapper is static. This will make it really difficult to mock it for the purpose of unit testing (or any other purpose)
There should be no logic in the controller.
Read this for example: Where should I put my controller business logic in MVC3
or this one: https://softwareengineering.stackexchange.com/questions/165444/where-to-put-business-logic-in-mvc-design
I know it's tempting but the less logic you put there the best you will find yourself in the future.
the answer in my opinion is:
You should read your property in a business layer benhead the controller and pass it all the way up to the view in a model object.
I agree with Maurizio in general that all business logic should be in some service/business logic layer. However in this case since you're only fetching a value from web.config whether, in your controller action, you do:
var someValue = App.TaxBillLink;
or you do:
var someValue = _linkService.GetTodaysLink();
really doesn't matter much unless there is some sort of logic there that needs to be unit tested.
I have a view model that represents all the fields available for searching. I'd like to add some logic that would be able to identify if the search values are all the same and determine whether to hit the DB again for their query.
I think I would have to do something like..
after user submits form save form values to some
temporary field.
upon second submission compare temp value to form values collection.
if values are equal set property in view
model IsSameSearch = true
I'd like to use the Post Redirect Get Pattern too. So that My search View doesn't do anything except post the form values to another action that processes and filters the data, which is then "Getted" using Ajax.
The SearchViewModel contains many many search parameters. Here is an abbreviated version.
public bool UseAdvancedSearch { get; set; }
public bool isSameSearch { get; set; }
/// <summary>
/// Gets or sets the page.
/// </summary>
[HiddenInput]
[ScaffoldColumn(false)]
public int Page { get; set; }
[HiddenInput]
[ScaffoldColumn(false)]
public string SortOption { get; set; }
/// <summary>
/// Gets or sets the address keywords.
/// </summary>
[Display(Name="Address")]
public string AddressKeywords { get; set; }
/// <summary>
/// Gets or sets the census.
/// </summary>
public string Census { get; set; }
/// <summary>
/// Gets or sets the lot block sub.
/// </summary>
public string LotBlockSub { get; set; }
/// <summary>
/// Gets or sets the owner keywords.
/// </summary>
[Display(Name="Owner")]
public string OwnerKeywords { get; set; }
/// <summary>
/// Gets or sets the section township range.
/// </summary>
public string SectionTownshipRange { get; set; }
/// <summary>
/// Gets or sets the strap.
/// </summary>
///
[Display(Name="Account Number/Parcel ID")]
public string Strap { get; set; }
/// <summary>
/// Gets or sets the subdivision.
/// </summary>
public string Subdivision { get; set; }
/// <summary>
/// Gets or sets the use code.
/// </summary>
[Display(Name = "Use Code")]
public string UseCode { get; set; }
/// <summary>
/// Gets or sets the zip code.
/// </summary>
[Display(Name="Zip Code")]
public string ZipCode { get; set; }
If you are getting data from Entity Framework you could cache the data at EF level. Look at the package entity framework extended https://github.com/loresoft/EntityFramework.Extended. It is as simple as adding method .FromCache () to the query you use to retrieve and filter the data and it will cache the query result. Make sure you load all the data required using includes etc.
You wouldn't have to worry about same search in model as the caching provider would look at filter settings and determine that it was different. Alternatively cache the data before filtering and then filter the cached results. This is more appropriate if you have lots of filter parameters with significant variance as you will only have to cache 1 large result rather than thousands of smaller results.
You can get more advanced and specify cache period e.g. Cache for 10 minutes
What you are describing is called caching.
One way to accomplish that in your scenario would be to implement GetHashCode() in a way that it would take into account all your fields/properties to compute a unique value. That way you can use your Hash as the key entry in your cache, and store the results with that key.
For that actual caching you could just use the MemoryCache class provided by the .Net Framework if you are not deploying to a web farm.
Also, if you are familiar with IoC and DI (such as using Unity), things like this can be implemented as an Interceptor, and only requiring you to add an attribute to the method you'd like to cache. That way you implement caching only once as a cross-cutting concern and not fill up your application code with things like this.