I am a beginner to coding in U-SQL/C#. I am stuck in a place during windowing/aggregation.
My Data looks like
Name Date OrderNo Type Balance
one 2018-06-25T04:55:44.0020987Z 1 Drink 15
one 2018-06-25T04:57:44.0020987Z 1 Drink 70
one 2018-06-25T04:59:44.0020987Z 1 Drink 33
one 2018-06-25T04:59:49.0020987Z 1 Drink 25
two 2018-06-25T04:55:44.0020987Z 2 Drink 22
two 2018-06-25T04:57:44.0020987Z 2 Drink 81
two 2018-06-25T04:58:44.0020987Z 2 Drink 33
two 2018-06-25T04:59:44.0020987Z 2 Drink 45
In U-SQL I am adding a unique id based on combinations of name, orderno and type and for the purpose of sorting, I am adding another one including the date.
#files =
EXTRACT
name string,
date DateTime,
type string,
orderno int,
balance int
FROM
#InputFile
USING new JsonExtractor();
#files2 =
SELECT *,
DENSE_RANK() OVER(ORDER BY name,type,orderno,date) AS group_id,
DENSE_RANK() OVER(ORDER BY name,type,orderno) AS id
FROM #files;
My Data now looks like this:
Name Date OrderNo Type Balance group_id id
one 2018-06-25T04:55:44.0020987Z 1 Drink 15 1 1
one 2018-06-25T04:57:44.0020987Z 1 Drink 70 2 1
one 2018-06-25T04:59:44.0020987Z 1 Drink 33 3 1
one 2018-06-25T04:59:49.0020987Z 1 Drink 25 4 1
two 2018-06-25T04:55:44.0020987Z 2 Drink 22 5 2
two 2018-06-25T04:57:44.0020987Z 2 Drink 81 6 2
two 2018-06-25T04:58:44.0020987Z 2 Drink 33 7 2
two 2018-06-25T04:59:44.0020987Z 2 Drink 45 8 2
(I have added only 4 records per group but there are multiple per group)
I am stuck at determining the difference between successive rows in the balance column in each group.
Expected Output for Part 1:
Name Date OrderNo Type Balance group_id id increase
one 2018-06-25T04:55:44.0020987Z 1 Drink 15 1 1 0
one 2018-06-25T04:57:44.0020987Z 1 Drink 70 2 1 55
one 2018-06-25T04:59:44.0020987Z 1 Drink 33 3 1 -37
one 2018-06-25T04:59:49.0020987Z 1 Drink 25 4 1 -8
two 2018-06-25T04:55:44.0020987Z 2 Drink 22 5 2 0
two 2018-06-25T04:57:44.0020987Z 2 Drink 81 6 2 59
two 2018-06-25T04:58:44.0020987Z 2 Drink 33 7 2 -48
two 2018-06-25T04:59:44.0020987Z 2 Drink 45 8 2 8
For every new group (defined by id) the increase should start from zero.
I went through stack overflow and saw the lag function from transgresql. I could not find a C# equivalent. Is that applicable in this case?
Any help is appreciated. Further clarification will be provided if required.
Update: When I use CASE WHEN my solution looks like this
CURRENT OUTPUT DESIRED OUTPUT
id Balance Increase id Balance Increase
1 15 0 1 15 0
1 70 55 1 70 55
1 33 -37 1 33 -37
1 25 -8 1 25 -8
2 22 "-3" 2 22 "0"
2 81 59 2 81 59
2 33 -48 2 33 -48
2 45 12 2 45 12
Look at the highlighted row. The increase column must start at 0 for each id.
Update: I was able to solve the first part of my question. See my answer below.
The second part that I had posted earlier was incorrectly posted. I have removed that.
You can try to use LAG window function get previous Balance in a subquery, then use where write the condition.
SELECT * FROM (
SELECT *,
DENSE_RANK() OVER(ORDER BY name,type,orderno,date) AS group_id,
DENSE_RANK() OVER(ORDER BY name,type,orderno) AS id,
(CASE WHEN LAG(Balance) OVER(ORDER BY name,type,orderno) IS NULL THEN 0
ELSE Balance - LAG(Balance) OVER(ORDER BY name,type,orderno)
END) as increase
FROM #files
) t1
WHERE increase > 50
The query that finally worked for me was this..
#files =
EXTRACT
name string,
date DateTime,
type string,
orderno int,
balance int
FROM
#InputFile
USING new JsonExtractor();
#files2 =
SELECT *,
DENSE_RANK() OVER(ORDER BY name,type,orderno) AS group_id
FROM #files;
#files3 =
SELECT *,
DENSE_RANK() OVER(PARTITION BY group_id ORDER BY date) AS group_order
FROM #files2;
#files4 =
SELECT *,
(CASE WHEN group_order == 1 THEN 0
ELSE balance - LAG(balance) OVER(ORDER BY name,type,orderno)
END) AS increase
FROM #files3;
I have a table like this to be exported to Excel
name date value1 value2 value 3
A 09/09/2015 5 10 2
B 09/09/2015 6 6 22
C 09/09/2015 4 3 11
A 10/09/2015 15 1 2
B 10/09/2015 6 16 27
C 10/09/2015 4 31 11
A 11/09/2015 15 1 2
B 11/09/2015 6 16 27
C 11/09/2015 4 31 11
can we pivot this to something like this (using SQL or C# datatable)
09/09/2015 | 10/09/2015 | 11/09/2015
value1 value2 value3 | value1 value2 value3 | value1 value2 value3
A 5 10 12 | 15 1 2 | 15 1 2
B 6 6 22 | 6 16 27 | 6 16 27
C 4 3 11 | 4 31 11 | 4 31 11
since your exporting this to Excel why not sumarize the table in Excel into a Pivottable and from there put the name column into the row box, date column into column box, the values into the value box.
After that you can insert some nice diagrams and then you have a nicer view of the data if you ask me.
I need to make a query that select all rows belong to a ID from the datatable where Lesson from that ID has two specific values. For example ID 2 and 3 have lessons D and E that I want to take all the rows of ID 2 and ID 3.
My data is in MS Access.
MY Programming language is C#.
I already try to write some a query
"SELECT * FROM MainData WHERE ID IN ( SELECT ID FROM MainData GROUP BY ID HAVING (Lesson ='" + txtbox_startsubject.Text + "' and '" + TargetPoint + "'))";
The format of data:
ID Lesson Time Score
1 C 165 4
1 E 190 3
1 H 195 3
1 I 200 4
2 A 100 2
2 B 150 5
2 D 210 2
2 R 10 3
2 E 110 4
3 D 130 5
3 E 190 5
3 H 210 4
3 I 160 4
3 J 110 4
4 E 120 3
4 H 150 4
4 J 170 4
I have this datatable in c#
Date Employee Job1 Job2 Job3
1/1/2012 a 1 1 1
1/1/2012 b 2 2 2
1/1/2012 c 2 1 4
1/1/2012 d 4 2 1
1/2/2012 a 3 2 5
1/2/2012 b 2 2 2
1/2/2012 c 3 3 3
1/2/2012 d 1 1 1
1/3/2012 a 5 5 5
1/3/2012 b 2 2 6
1/3/2012 c 1 1 1
1/3/2012 d 2 3 4
2/1/2012 a 2 2.5 2
2/1/2012 b 5 5 2
2/1/2012 c 2 2 2
2/2/2012 a 3 3 3
2/2/2012 b 2 3 3
3/1/2012 a 4 4 11
3/5/2012 a 14 42.5 15
3/6/2012 a 21 12.143 22
3/8/2012 a 8.9 45 27
3/8/2012 b 4.4 25 31
I want to loop through the values monthwise such that i can store the value in a datatable and will do the other calculation with that. Here with this example it will have three datatable, first with january values, another one with feb and last one with march rows. How can this be done by Linq. Please suggest the Linq syntax which will group the results month wise.
var monthEmpGroups = dt.AsEnumerable()
.Select(r => new
{
Year = DateTime.Parse(r.Field<string>("DATE")).Year,
Month = DateTime.Parse(r.Field<string>("DATE")).Month.ToString()
})
.GroupBy(x => x.Month);
var dtf = CultureInfo.CurrentCulture.DateTimeFormat;
foreach (var empGroup in monthEmpGroups)
{
int month = int.Parse(empGroup.Key);
string colName = dtf.GetMonthName(month);
// Here i want to get all the rows where month is colName (i.e.january, feb, march)
}
Please suggest if there is other way around to get the values month wise using LINQ.
You should move getting items for every group into monthEmpGroups query:
var monthEmpGroups = (from r in dt.AsEnumerable()
let month = DateTime.Parse(r.Field<string>("DATE")).Month
group r by month into g
select new { Month = g.Key, Items = g.ToList() });
With that you can easily get desired results:
var dtf = System.Globalization.CultureInfo.CurrentCulture.DateTimeFormat;
foreach (var empGroup in monthEmpGroups)
{
int month = empGroup.Month;
string colName = dtf.GetMonthName(month);
List<DataRow> items = empGroup.Items;
}
I am looking to update a calculated sum in sql
Basically I have a table:
ImportID SeiralNumber Day Hour value Difference Complete
1 123 1 1 6 NULL 0
2 123 1 2 8 NULL 0
3 123 1 5 21 NULL 0
4 123 1 6 28 NULL 0
5 222 2 2 12 NULL 0
6 222 2 5 18 NULL 0
7 222 2 4 16 NULL 0
8 222 1 12 8 NULL 0
For each serial number there will be a day 1-365 and hour through 1-12, all I want to do is calculate the difference filed from the record before
So take ImportID 6, I need to get the record which is on the same day and the hour before (importID 7) then I need to update the Difference using the value field which is 18 -17 = 1.
N.B. There may be gaps in the sequence and if there is no previous record then the difference should stay as NULL. Once they have been calculated they need to be inserted into a new table only when the difference is now not null and it doesn't exist in the table already, on a successful insert they get marked as complete. Also a record before can be a previous day (day 1 hour 12) is the record before (day 2, hour 1)
Currently I am using a loop to select the null values, get the previous record, update the record, if its OK insert into other table, update the Completed field.
My issue is that this is working on a million records and it is taking a long while to Select the applicable records (completed = 0) into a temp table and loop through each.
Is there any quicker way to mass process these as an update statement? Or separate statements?
The result should be
ImportID SeiralNumber Day Hour value Difference Complete
1 123 1 1 6 NULL 0
2 123 1 2 8 2 1
3 123 1 5 21 NULL 0
4 123 1 6 28 7 1
5 222 2 1 12 4 1
6 222 2 5 18 2 1
7 222 2 4 16 NULL 0
8 222 1 12 8 NULL 0
Thanks in advance
I think this is basically it isn't it?
DECLARE #TABLE TABLE
(
ImportId INT,
SerialNumber INT,
Day INT,
Hour INT,
Value INT,
Difference INT,
Complete INT
)
INSERT INTO #TABLE VALUES
(1,123,1,1,6,NULL,0),
(2,123,1,2,8,NULL,0),
(3,123,1,5,21,NULL,0),
(4,123,1,6,28,NULL,0),
(5,222,2,1,12,NULL,0),
(6,222,2,5,18,NULL,0),
(7,222,2,4,16,NULL,0),
(8,222,1,12,8,NULL,0)
SELECT * FROM #Table
UPDATE T
SET T.Difference = T.Value - TT.Value,
Complete = 1
FROM
(
SELECT *
,ROW_NUMBER() OVER (PARTITION BY SerialNumber ORDER BY Day ASC, Hour ASC) AS RowCounter
FROM #TABLE
WHERE Complete = 0 --Ignore completed ones
)AS T
INNER JOIN
(
SELECT *
,ROW_NUMBER() OVER (PARTITION BY SerialNumber ORDER BY Day ASC, Hour ASC) AS RowCounter
FROM #TABLE
)AS TT
ON T.SerialNumber = TT.SerialNumber
WHERE
(
T.RowCounter = TT.RowCounter + 1
AND
T.Day = TT.Day
AND
T.Hour = TT.Hour + 1
)
OR
(
T.Day = TT.Day + 1
AND
T.Hour = 1
AND
TT.Hour = 12
)
SELECT * FROM #TABLE