I am using MariaDB. I have a table that I create for every IoT device at the time of the first insertion with a stored procedure. If anyone wonders Why I create a new table for every device is devices publish data every 5 seconds and it is impossible for me to store all of them in a single table.
So, my table structure is like below:
CREATE TABLE IF NOT EXISTS `mqttpacket_',device_serial_number,'`(
`data_type_id` int(11) DEFAULT NULL,
`data_value` int(11) DEFAULT NULL,
`inserted_date` DATE DEFAULT NULL,
`inserted_time` TIME DEFAULT NULL,
FOREIGN KEY(data_type_id) REFERENCES datatypes(id),
INDEX `index_mqttpacket`(`data_type_id`,`inserted_date`)) ENGINE = INNODB;
I have a very long SELECT query like below to fetch the data between selected type, date, and time.
SELECT mqttpacket_123.data_value, datatypes.data_name, datatypes.value_mult,
CONCAT(mqttpacket_123.inserted_date, ' ',
mqttpacket_123.inserted_time) AS 'inserted_date_time'
FROM mqttpacket_123
JOIN datatypes ON mqttpacket_123.data_type_id = datatypes.id
WHERE mqttpacket_123.data_type_id IN(1,2,3,4,5,6)
AND CASE WHEN mqttpacket_123.inserted_date = '2021-11-08'
THEN mqttpacket_123.inserted_time > '12:25:00'
WHEN mqttpacket_123.inserted_date = '2021-11-15'
THEN mqttpacket_123.inserted_time< '12:25:00'
ELSE (mqttpacket_123.inserted_date BETWEEN '2021-11-08'
AND '2021-11-15')
END;
and this returns around 500k records of the sample below:
| data_value | data_name | value_mult | inserted_date_time |
--------------------------------------------------------------------------------
| 271 | name_1 | 0.1 | 2021-11-08 12:25:04 |
| 106 | name_2 | 0.1 | 2021-11-08 12:25:04 |
| 66 | name_3 | 0.1 | 2021-11-08 12:25:04 |
| 285 | name_4 | 0.1 | 2021-11-08 12:25:04 |
| 61 | name_5 | 0.1 | 2021-11-08 12:25:04 |
| 454 | name_6 | 0.1 | 2021-11-08 12:25:04 |
| 299 | name_7 | 0.1 | 2021-11-08 12:25:04 |
Affected rows: 0 Found rows: 395,332 Warnings: 0 Duration for 1 query: 0.734 sec. (+ 7.547 sec. network)
I keep only the last 2 weeks' data in my tables and clean up the previous data as I have a backup system.
However, Loading the query result to DataTable also takes ~30sec. which is 4 times slower than MySQL.
Do you have any suggestions to improve this performance?
PS. I call this query from C# by the following statement in a Stored Procedure of RunQuery which takes the query and performs it as it is.
public DataTable CallStoredProcedureRunQuery(string QueryString)
{
DataTable dt = new DataTable();
try
{
using (var conn = new MySqlConnection(_connectionString))
{
conn.Open();
using (var cmd = new MySqlCommand("SP_RunQuery", conn))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#query_string", MySqlDbType.VarChar).Value = QueryString;
using (MySqlDataAdapter sda = new MySqlDataAdapter(cmd))
{
sda.Fill(dt);
}
}
}
}
catch (Exception ex)
{
IoTemplariLogger.tLogger.EXC("Call Stored Procedure for RunQuery failed.", ex);
}
return dt;
}
EDIT: My sensors push a single MQTT packet which contains ~50 different data. There are 12 times 5seconds in a minute. So, basically, I receive ~600 rows per minute per device.
Data insertion is done in a Stored Procedure async. I push the JSON content along with the device_id and I iterate on the JSON to parse and insert into the table.
PS. The following code is just for clarification. It works fine.
/*Dynamic SQL -- IF they are registered to the system but have notable, create it.*/
SET create_table_query = CONCAT('CREATE TABLE IF NOT EXISTS `mqttpacket_',device_serial_number,'`(`data_type_id` int(11) DEFAULT NULL, `data_value` int(11) DEFAULT NULL,`inserted_date` DATE DEFAULT NULL, `inserted_time` TIME DEFAULT NULL, FOREIGN KEY(data_type_id) REFERENCES datatypes(id), INDEX `index_mqttpacket`(`data_type_id`,`inserted_date`)) ENGINE = InnoDB;');
PREPARE stmt FROM create_table_query;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
/*Loop into coming value array. It is like: $.type_1,$.type_2,$.type_3, to iterate in the JSON. We reach each value like $.type_1*/
WHILE (LOCATE(',', value_array) > 0)
DO
SET arr_data_type_name = SUBSTRING_INDEX(value_array,',',1); /*pick first item of value array*/
SET value_array = SUBSTRING(value_array, LOCATE(',',value_array) + 1); /*remove picked first item from the value_array*/
SELECT JSON_EXTRACT(incoming_data, arr_data_type_name) INTO value_iteration; /*extract value of first item. $.type_1*/
SET arr_data_type_name := SUBSTRING_INDEX(arr_data_type_name, ".", -1); /*Remove the $ and the . to get pure data type name*/
/*Check the data type name exists or not in the table, if not insert and assign it's id to lcl_data_type_id*/
IF (SELECT COUNT(id) FROM datatypes WHERE datatypes.data_name = arr_data_type_name) > 0 THEN
SELECT id INTO lcl_data_type_id FROM datatypes WHERE datatypes.data_name = arr_data_type_name LIMIT 1;
ELSE
SELECT devices.device_type_id INTO lcl_device_type FROM devices WHERE devices.id = lcl_device_id LIMIT 1;
INSERT INTO datatypes (datatypes.data_name,datatypes.description,datatypes.device_type_id,datatypes.value_mult ,datatypes.inserted_time) VALUES(arr_data_type_name,arr_data_type_name,lcl_device_type,0.1,NOW());
SELECT id INTO lcl_data_type_id FROM datatypes WHERE datatypes.data_name = arr_data_type_name LIMIT 1;
END IF;
/*To retrieve the table of which device has which datatypes inserted, this is to not to retrieve the datatypes unneccesseraly for the selected device*/
IF (SELECT COUNT(device_id) FROM devicedatatypes WHERE devicedatatypes.device_id = lcl_device_id AND devicedatatypes.datatype_id = lcl_data_type_id) < 1 THEN
INSERT INTO devicedatatypes (devicedatatypes.device_id, devicedatatypes.datatype_id) VALUES(lcl_device_id,lcl_data_type_id);
END IF;
SET lcl_insert_mqtt_query = CONCAT('INSERT INTO mqttpacket_',device_serial_number,'(data_type_id,data_value,inserted_date,inserted_time) VALUES(',lcl_data_type_id,',',value_iteration,',''',data_date,''',''',data_time,''');');
PREPARE stmt FROM lcl_insert_mqtt_query;
EXECUTE stmt;
SET affected_data_row_count = affected_data_row_count + 1;
END WHILE;
Here and here are also extra information that can be found of the server and database regarding the comments.
I have an SSD on the server. There is nothing important else that works other than my dotnet application and database.
It is usually better to have a DATETIME column instead of splitting it into two (DATE and TIME) columns. That might simplify the WHERE clause.
Having one table per device is usually a bad idea. Instead, add a column for the device_id.
Not having a PRIMARY KEY is a bad idea. Do you ever get two readings in the same second for a specific device? Probably not.
Rolling those together plus some other likely changes, start by changing the table to
CREATE TABLE IF NOT EXISTS `mqttpacket`(
`device_serial_number` SMALLINT UNSIGNED NOT NULL,
`data_type_id` TINYINT UNSIGNED NOT NULL,
`data_value` SMALLINT NOT NULL,
`inserted_at` DATETIME NOT NULL,
FOREIGN KEY(data_type_id) REFERENCES datatypes(id),
PRIMARY KEY(device_serial_number, `data_type_id`,`inserted_at`)
) ENGINE = INNODB;
That PK will make the query faster.
This may be what you are looking for after the change to DATETIME:
AND inserted_at >= '2021-11-08 12:25:00'
AND inserted_at < '2021-11-08 12:25:00' + INTERVAL 7 DAY
To keep 2 weeks' worth of data, DROP PARTITION is an efficient way to do the delete. I would use PARTITION BY RANGE(TO_DAYS(inserted_at)) and have 16 partitions, as discussed in http://mysql.rjweb.org/doc.php/partitionmaint
If you are inserting a thousand rows every 5 seconds -- With table-per-device, you would need a thousand threads each doing one insert. This would be a nightmare for the architecture. With a single table (as I suggest), and if you can get the 1000 rows together in a process at the same, time, do one multi-row INSERT every 5 seconds. I discuss other high speed ingestion.
Rate Per Second = RPS
Suggestions to consider for your instance [mysqld] section
innodb_io_capacity=500 # from 200 to use more of available SSD IOPS
innodb_log_file_size=256M # from 48M to reduce log rotation frequency
innodb_log_buffer_size=128M # from 16M to reduce log rotation avg 25 minutes
innodb_lru_scan_depth=100 # from 1024 to conserve 90% CPU cycles used for function
innodb_buffer_pool_size=10G # from 128M to reduce innodb_data_reads 85 RPS
innodb_change_buffer_max_size=50 # from 25 percent to expedite pages created 590 RPhr
Observation,
innodb_flush_method=O_DIRECT # from fsync for method typically used on LX systems
You should find these significantly improve task completion performance. View profile for free downloadable Utility Scripts to assist with performance tuning.
There are additional opportunities to tune Global Variables.
I have a table column sales_price of type money. When I am retrieving its value through a stored procedure and the value is inserted in the datatable dtDetail which is later shown in a datagrid dataListDetail (from a WinForm), it returns the value up to 4 decimal places (because of the type money). I need to set the value up to two decimal places.
private void createTable()
{
this.dtDetail.Columns.Add("sales_price", System.Type.GetType("System.Decimal"));
this.dataListDetail.DataSource = this.dtDetail;
this.dataListDetail.Columns["sales_price"].HeaderText = "Sales Price";
}
After inserting a value like 50 or 50,5 I expect for the sales_price value to display in the datagrid with only 2 decimal places (Ex. 50,00 or 50,50), but it ends displaying it with 4 decimal places (Ex. 50,0000 or 50,5000) by default (due to the type Money).
Thank you in advance!
Just set Format on DefaultCellStyle:
this.dataListDetail.Columns["sales_price"].DefaultCellStyle.Format = "C2";
Maybe .ToString("c2") is what you want? It will convert to your systems currency format.
I'm running a 3-node Cassandra 3.0.0 cluster running on AWS EC2 i3.large instances and I've been playing around with using the C# driver for Cassandra. Executing the following query (which is very simple) takes approximately 300 ms (to scan the single partition and return the top 100 rows).
var rs = session.Execute("SELECT col1, col6, col7 FROM breadcrumbs WHERE col1='samplepk' LIMIT 100;");
My data model is:
Column 1 = a 13-character string
Column 2 = a 23-character string
Column 3 = a date/time timestamp
Column 4 = a 4-digit integer
Column 5 = a 3 digit integer
Column 6 = a latitude value
Column 7 = a longitude value
Column 8 = a 15-digit double
Column 9 = a 15-digit double
I defined my primary key as Col1, col2.
My C# driver code is as follows:
Cluster cluster = Cluster.Builder().AddContactPoint(~~~~~IP Here~~~~).Build();
ISession session = cluster.Connect(~~~keyspacename~~~);
long ticks = DateTime.Now.Ticks;
var rs = session.Execute("SELECT col2, col6, col7 FROM breadcrumbs WHERE partitionkey=~targetkey~ LIMIT 100;");
Console.WriteLine((DateTime.Now.Ticks - ticks)/Math.Pow(10,4)+" ms");
Console.ReadKey();
Is that abnormally slow or are my expectations too high? If it is slow, does anyone have any ideas about what's causing it?
If I forgot to provide any pertinent details, please leave a comment :) .
Thanks in advance.
I have a datatable with a few columns .
I am trying to add the column values using the datacolumn.expression.
The columns used for adding is of type decimal. Also the calculated column is also decimal. But while processing the expression, (like datatable column1+ datatable column2) its just appending the data.
SlNo Name F1 F2 F3
1 A 1 2 3
2 B 3 4 5
I am expecting an output similar to this.
SlNo Name F1 F2 F3 Total
1 A 1 2 3 6
2 B 3 4 5 12
What I tried.
dtTempData.Columns.Add("Total", typeof(Decimal));
dtTempData.Columns["Total"].Expression = "[F1]+[F2]+[F3]";
Now the output I am getting is in the following way
123
345
its just appending the data.Thanks in advance of any help.
I don't know why this is happening.
dtTempData.Columns.Add("Total", typeof(Decimal));
dtTempData.Columns["Total"].DefaultValue = 0;
dtTempData.Columns["Total"].Expression = expression;
This is the way I created the columns, but while performing suming based on expression, it appends the data.
I am importing data from another datatable which is type of string. So I tried to convert the data by using the following manner.
string expression="Convert(F1, 'System.Decimal') + Convert(F2,
'System.Decimal') + Convert(F3, 'System.Decimal')"
Now this is working and the Total column is having the value after addition. Thanks all for your help.
Can I find allowed variable length for nvarchar type in sql server 2008 from c# code in asp.net application?
For eg:
nvarchar(?)
I want to find the maximum allowed number for "?" from c# code.
You can use this T-SQL query to look at the system catalog views:
SELECT
[max_length]
FROM sys.columns
WHERE [object_id] = OBJECT_ID('YourTableNameHere')
AND name = 'YourColumnNameHere'
This will return the stored, defined maximum length (in characters) for your column
Update: if you want to find out the max length of a type (not a column of any of your tables), you can use this query instead:
SELECT
name, max_length
FROM sys.types
WHERE name IN ('varchar', 'nvarchar')
Be aware: this returns the max length in bytes (not in characters!) so you get 8000 for both types. For varchar, 8000 bytes is equal to 8000 characters, while for nvarchar, 8000 bytes corresponds to 4000 characters.
Please using TableSchema method for getting all details of column .
SqlDataReader reader= command.ExecuteReader();
using (var schemaTable = reader.GetSchemaTable())
{
foreach (DataRow row in schemaTable.Rows)
{
string ColumnName= row.Field<string>("ColumnName");
string DataTypeName= row.Field<string>("DataTypeName");
short NumericPrecision= row.Field<short>("NumericPrecision");
short NumericScale= row.Field<short>("NumericScale");
int ColumnSize= row.Field<int>("ColumnSize");
Console.WriteLine("Column: {0} Type: {1} Precision: {2} Scale: {3} ColumnSize {4}",
ColumnName, DataTypeName, NumericPrecision, scale,ColumnSize);
}
}
Thanks .
You can use the .Size property of the SqlParameter:
For output parameters with a variable length type (nvarchar, for example), the size of the parameter defines the size of the buffer holding the output parameter. The output parameter can be truncated to a size specified with Size. For character types, the size specified with Size is in characters.
See MSDN Documentation.
If you mean from a database field, you can get the data with the .GetSchemaTable() method from the DataReader object. The ColumnSize attribute has the information you need.
Read this article to see how to do it.