I'm working on an AWS serverless application, I'm new to AWS so please correct any misconceptions in this question. We have around 30 lambda functions doing basic CRUD operations on a SQL database hosted in RDS. We used EntityFramework Core to create the database. This means that, in order to use the database, the lambda functions each need access to the EF connection string. I didn't want to leave the connection string in memory (it contains a plaintext password) so I put it in an encrypted environment variable.
The only way I could work out how to do this was to add an encrypted environment variable through the lambda management console GUI to every lambda function individually. This was tedious but it worked. I've now changed the solution a bit and we require a different set of lambda functions, these need environment variables adding to them too and I don't want to do it manually again.
My question:
Is there an easy way to add encrypted environment variables (or something similar) that can be accessed by all of my lambda functions? Is there a method to add them in bulk? A higher-level variable that I can use?
I have tried to find information in the Amazon docs (here for example) but had little success.
Lambda is only concerned about getting the environment variables that it needs and it is not responsible for centrally managing environment variables.
What you would need to do is to handle your environment variables in your deployment process.
How do you currently deploy your 30 lambda functions?
If you use CodeBuild, it can access AWS EC2 Parameter Store where you can centrally store and manage your sensitive environment variables.
Other CI/CD tools also have their own way of centrally managing environment variables.
Update:
You can use the AWS EC2 Parameter Store programmatically. This means that you can retrieve your stored environment variables from inside your Lambda function during startup so you don't need to do this at deployment time.
Doing it on deployment time is still better if that is possible in your use case.
I would not call this as a solution rather a work around to address your problem.
You can configure your environment variables and store it onto S3 bucket.
And using S3 client within your each lambda , you can read specific variable you are interested in.
Hope this helps.
I would suggest to add ConnectionString at AWS console and to not use production connection string at config file to avoid those kinds of problems. Log into AWS console, go to lambda function and add connection string as an environment variable.
PS. In case you will struggle with syntax. If your json config file looks like this:
{
"Settings":
{
"ConnectionString":"",
"SomethingElse"
}
}
Use this syntax to configure connection string:
Settings__ConnectionString
I am late to the party. How about saving your connection string into an DynamoDB table and let each lambda take from it. I assume one connection string for all of it. Most of the time for SQL Server.
Related
Currently we have set up our UI test automation in one environment(Dev). We are using C#, .Net, Visual Studio, specflow and MSTest
CONFIG
We read an app.config file for environment specific variables. We use Azure Pipeline for CI and run these tests on a nightly build.
<configuration>
<appSettings>
<add key="DevApp" value="https://dev.websitename.com />
</appSettings>
<connectionStrings>
<add name="DevDatabase" connectionString="http://dev.url/" />
</connectionStrings>
</configuration>
Now we want to run these tests on our UAT environment as well. The setup will be the same, and we want to run the same tests with the same data. The only difference is that for UAT we will point to different URL's and a different database.
for example
Dev env = https://dev.websitename.com
UAT env = https://uat.websitename.com
server name="DevDatabase" connectionString="http://dev.url/"
server name="UATDatabase" connectionString="http://uat.url/"
PASSWORDS
In terms of password, out application is an internal application and we use windows auth. So in our dev and uat environment we have the same password set up for all users. So in Dev = devpassword and UAT = uatpassword
For both dev and test we are using the same users with the password being the only difference.When testing we launch the browser using impersonation and launch the browser as 'run as' for that user
var service = ChromeDriverService.CreateDefaultService(driverpath)
if user is not null then we do this
var pwd = new SecureString()
service.StartDomain = Configurationhelper.Domain
service.StartupUserName = username
service.StartupPassword= = pwd
service.StartupLoadUserProfile = true
we store domain and password and other environmental variables in a separate config file as constants.
**Main issue: **
This wont work now so I think it could be best to store passwords as secrets in AZURE pipeline variables? if so, how would i change this code? for example
The server team, db team and devops team have taken care of server,db setup and urls etc
So for me its just configuring the test automation repo with my changes to configuration
What could be an elegant approach to do this?
AZURE PIPELINE
How could we run tests for both these environments in parallel? by parallel i mean having both run on a nightly run. Our Azure pipeline has 2 separate clients UAT and DEV pointing to the same artifact. The tasks and Variable are the same for both environments but with different values obviously
Currently they both would run in isolation
Solutions to this problem come down to how does the context (in your case the environment and all its associated connection strings and URLs) get to the tests where they will be consumed. In your question, you stated several orthogonal concerns:
using the same data
running in a different environment
running in parallel
Not mentioned is another concern
how to handle secrets (e.g. passwords in connection strings)
I'll explain one solution (a strategy really) that addresses these concerns, and why it appears to be a maintainable and extensible solution.
Using the same data
This can be very simple or very complex. The simple solution is to create a database of canonical and representative test data, and to then swap in that database to your environment. That can be done via your database's backup/restore or by creating the data programmatically. You would need to have a mechanism to restore or wipe the data whether the test(s) succeeds or fails.
Very rarely will using the environment's database "as is" lead to reliable tests; tests often modify state, and a database is the ultimate form of state; mutations of state will affect future tests.
It is with this last sentence that a full swap that occurs before/after each test is probably a) faster (occurring at a bulk/macro level with a quicker swap function), b) more maintainable (data can be reviewed/created ahead of time) and c) less brittle.
Running in a different environment
This, like the heart of your question discusses, is where you come down to whether to use multiple files or a single file. Using multiple files means you can take advantage of some of the built-in .NET configuration mechanisms that allow you to specify the environment. That means duplicating the files and changing the values to reflect the environment.
The other way, you mentioned, is storing all of this information into a single configuration file. If you do it this way, you need some sort of discriminator to disambiguate the entries, and your test needs to be able to pass in the environment name to some API to pull the value. I prefer this mechanism personally, because when you add a new feature/test you can add all the configuration in one place.
So the two mechanisms are roughly the same in terms of work, except the latter leads to a more compact editing session when adding new config, and from a readability/maintainability scenario, you have fewer places to look.
Secrets
If you follow the single-source of configuration approach, you simply extend that to your secrets, but you select the appropriate secret store (e.g. a secrets file or Azure Key Vault, or some such... again with an environment-based discriminator). Here's an example:
{
"DEV.Some.Key" : "http://devhost/some/path",
"UAT.Some.Key" : "https://uathost/some/other/path"
...
}
Using a discriminator means far less changes to your DevOps pipeline, which is, from a developer/editing experience, most likely slower and more cumbersome than editing a file or key vault.
Running in parallel
While you could rotate out the context and design your solution to run in parallel using the MSTest mechanisms, it would be more elegant to allocate this to your pipeline itself, and have enough resources to be able to run these pipelines in parallel by having enough build agents and so on.
Conclusion
It comes down to which parts of the solution should be addressed by which resources. The solution above allocates environmental selection and test execution into the pipeline itself. Granular values such as connection strings and secrets are allocated to a single source to reduce the friction that occurs when having to edit these values.
Following this strategy might better leverage your team's skills as well. Those with a DevOps mindset can most likely spin up new environments and parallelize more readily than a Developer mindset, who would be more aware of what data needs to be setup and how to craft the tests.
I have done the following...
private static IDbConnectionProvider CreateSqlConnectionProvider(DbConfig dbConfig)
{
return new QcDbConnectionProvider(() =>
{
SqlConnectionStringBuilder csBuilder = new SqlConnectionStringBuilder();
if (!string.IsNullOrEmpty(dbConfig.DataSource))
csBuilder.DataSource = dbConfig.DataSource;
if (!string.IsNullOrEmpty(dbConfig.Database))
csBuilder.InitialCatalog = dbConfig.Database;
.
.
.
.
return new SqlConnection(csBuilder.ConnectionString);
});
}
The client is using VERACODE tool for doing code analysis and the VERACODE has detected a flaw "Untrusted initialization" at
return new SqlConnection(csBuilder.ConnectionString);
Also, the dbConfig is being initialized as shown below...
DbConfig configDbConfig = new DbConfig
{
Database = codeFile.ConfigurationDb,
DataSource = codeFile.DataSource,
IntegratedSecurity = sqlCredentials.UseWindowsAuthentication ? 1 : 0,
UserId = sqlCredentials.UseWindowsAuthentication ? null : sqlCredentials.SqlUserName,
ClearTextPassword = sqlCredentials.UseWindowsAuthentication ? null : sqlCredentials.SqlUserPassword
};
What else I need to do in order to fix this flaw? Also as per this link, I am creating the connection string using the SqlConnectionStringBuilder which is safe of creating the connection string.
Thanks in advance...
Description for Untrusted initialization issue is:
Applications should be reluctant to trust variables that have been initialized outside of its trust boundary. Untrusted initialization refers to instances in which an application allows external control of system settings or variables, which can disrupt service or cause an application to behave in unexpected ways. For example, if an application uses values from the environment, assuming the data cannot be tampered with, it may use that data in a dangerous way.
In your case you're reading data for dbConfig from file:
if (TryReadCodeFile(configurationProfileFile...)) {
DbConfig configDbConfig = new DbConfig...
}
Note that warning you get should also come with a line number (to circumscribe erroneous code). Almost everything in code you posted can generate this issue (I don't see where sqlCredentials comes from but it may even be another source of security problems if they're in clear text - or code to decrypt is accessible in your application).
From cited paragraph: "...application allows external control of system settings or variables, which can disrupt service...". This is the core of this issue: if your application uses external data without a direct control over them then its behavior can be changed modifying that data. What these external data are? List is all but not exhaustive:
Environment variables (for example to resolve a path to another file or program) because user may change them. Original files aren't touched but you read something else.
Paths (to load code or data) because user may redirect to something else (again original files aren't touched but you read something else).
Support files because user can change them (in your case, for example, to point to another server and/or catalog).
Configuration files because user can change them (same as above).
Databases because they may be accessible to other users too and they may be changed (but they may be protected).
How a malicious user may use this? Imagine each user is connected to a different catalog (according to their rule in organization). This cannot be changed and it's configured during installation. If they can have access to your configuration files they may change catalog to something else. They may also change DB host name to a tunnel where they may sniff data (if they have physical access to someone else's machine).
Also note that they also say "...assuming the data cannot be tampered with, it may use that data in a dangerous way". It means if, for example, your application runs on a web server and physical access is secured then you may consider that data safe.
Be aware your application will be secure as less secure item in your whole system. Note that to make an application safe (I know, this term is pretty vague) to encrypt password is not enough.
If support files may be manipulated then best thing you can do is to encrypt them with a public/private key encryption. A less optimal solution is to calculate a CRC or hash (for example) you'll apply to configuration files before you use them (they are able to change them but your application will detect this issue).
To summarize: you can ignore this issue but you have to prove your customer that data you rely on cannot be tampered. You can reasonably prove if at least one of these conditions is satisfied:
1) System where support files reside is not accessible by anyone else than your application. Your application security cannot be higher than system security.
2) Your support files are valid per-machine (to avoid copies between different machines) and they're encrypted in a way they cannot be changed (intentionally or not) by anyone.
3) Your support files are valid per-machine and they're hashed in a way your application can detect external changes.
4) It doesn't matter what users do with your configuration files, application itself cannot change its behavior because of that (for example it's a single installation where only one DB and one catalog exist).
The most important for connection strings is how they are stored. If they are stored in plaintext, this poses a security risk. So, it is advisable to store them in encrypted format and in application decrypt and use it.
I've tried searching but had no luck as I'm not sure I'm using the correct terminology.
I'm trying to figure out how to ask a user to input their server name on the first application run, store that and insert it into filepath/connection string.
Pseudocode for initial run:
I see this is the first time you ran this application. Please input your fileserver name.
user inputs: fileserver123x
Write Fileserver123x to text file.
Initialize database connection
string fileServername = read text file;
connection = new OleDbConnection(#"Provider=Microsoft.ACE.OLEDB.12.0;Data Source=\'"+fileServerName+#"'\share\folder\Database.accdb;Persist Security Info=False;");
rest of code
I'm looking to see if there is a better/more professional way to store the user file server information other than via text file and read it each time the DB connection is initialized.
Any thoughts?
Your concept looks fine, I'd suggest a few improvements:
Store the file server name in the Windows registry instead of a text file. This is where well-behaved Windows programs store their configuration data.
Store the complete path to the database instead of the file server name. That way, your customers don't have to use a fixed share/folder name.
Make a configuration window where this configuration data can be modified. This can also serve as the window you show on your first run.
Don't read the configuration data each time a connection is opened. Read it once when your application starts and store it in a global variable.
(Note: In general, global variables are a code smell, but storing global configuration data is usually considered a legitimate use case. If you want a more advanced solution that simplifies unit testing, look into dependency injection for your configuration data.)
I currently run MongoDB pointing to the appropriate data directory using the command line below:
mongod --dbpath "somePath/data"
But currently this is a manual step that I run before running a particular suite of tests. Is there a way I can set the path within the code (without calling a script or batch file) using the Mongo C# driver to use a specific data directory?
Update:
To clarify, the reason I'm looking to do this isn't for use in production code, but to isolate test databases for different test suites and to point at a disposable and isolated data directory so that each server instance is clean at the time of running tests and is only populated with the data it requires for the same server settings as production.
You probably won't find any way to do that. The Mongo C# Driver is for programming a MongoClient, not a server. The documentation for C# Driver for MongoDB says - MongoClient class serves as the root object for working with a MongoDB server. When you are programming a client, you automatically would assume that the server is up and running. Whether you do it manually or you write another code for it, that is a different story.
Very rarely would you allow people to connect to a machine and let them start a server AND A CLIENT on it. And why is it rare? You may try to start a server on another machine and screw up with that machine (which may be providing some other completely different service too!). There are some ways (and there are times when it is needed) to start a server remotely, but that is not what you can do using the MongoDB C# Driver.
Now, in order to get your task done, you can try this:
Start one mongod per database on your server, and make each mongod listen to a different port. Then in your code, you can connect your MongoClient to mongod running on the concerned database's port. You can achieve this by using a simple if condition (or a switch case) and checking what database the MongoClient wants to connect to and thus finding the right port value to put in the connection string. Each mongod can serve only one database or more or whatever you want.
So if you are running three mongod's on port1, port2 and port3 and all those three are connected to their respective db paths, the code can be somewhat like this:
var DBNAME = name_of_the_db;
string connectionString;
switch (DBNAME)
{
case name_of_first_DB:
connectionString = "mongodb://[user:pass#]hostname[:port1][/[DBNAME][?options]]";
break;
case name_of_second_DB:
connectionString = "mongodb://[user:pass#]hostname[:port2][/[DBNAME][?options]]";
break;
case name_of_third_DB:
connectionString = "mongodb://[user:pass#]hostname[:port3][/[DBNAME][?options]]";
break;
default:
Console.WriteLine("Invalid DB Name");
}
Answering the updated part of the question:
You can start mongod's on different partitions of the server. Even start the daemons from different drives altogether and make them listen to different ports. Goes without saying that the dbpaths should not be pointing to the same drive for any two databases to at least pretty closely mimic what you wanted.
Just to complete this answer I am adding what #Schaliasos has mentioned in comments.. Consider installing mongo as a window service.
I am designing a desktop application in C#, which needs to be connected to my online mysql database. I tried to give access in control panel a "%" which means from any IP, but it is not working. May be the hosting provider (bigrock) not allowing that.
Alternatively, I am trying to write some code in online on PHP which will get the "sql" as parameter and returns the output as JSON format using json_encode.
Is there any alternate methods which is better approach.
What error do you get when you try to connect? Timeout = firewalled; Permission denied = permissions not right etc.
One solution is to create a proxy with pre-coded queries (let's call then "stored procedures") - you can then say "Run query 5, parameters A, B and C". As this would be server-server (not public) you just need to add some basic authentication system (e.g. shared rotating key, checksum using parameters etc), but also ensure the queries are not dangerous if any parameters are thrown at it.
Disclaimer: It's a solution, but I'm not actually recommending that I'd do it unless you're very sure it's safe!
Do you have Cpanel ? If yes, then try adding your host in remote MySQL.
Here the link http://www.liquidweb.com/kb/enable-remote-mysql-connections-in-cpanel/ if you are unsure on how to do that.
I'd advise that you do not create a wildcard user that can connect to the database from anywhere using embedded MySQL credentials in your application. This is a bad idea.
It would be extremely easy to determine the credentials used by your application and then a malicious user could directly connect to your DB server and begin issuing queries to your database.
They will be able to issue SELECT statements for any information in your tables, even info they shouldn't see. It then becomes much easier to exploit any known or unknown vulnerabilities in MySQL much easier since now they have console access and can send data directly to the server. If the account has the DELETE privilege, they can erase all the data in your table(s).
Also, having a PHP script that issues queries provided by the application/end-user is not ideal because one can still freely issue queries. While that option is better than giving blanket access to a remote user, it is still a bad idea.
The way to go would be to identify all of the information that the C# application needs to access, and how, and write a simple web service/API that will receive parameters and issue its own queries and return the results using XML, JSON, or even SOAP. Try to abstract the database access as much as possible from the outside world for the best security.
Hope that helps.
I would do the following:
Create a user with the host of your public ip (www.whatismyip.com).
If that doesn't work, create a user with your host as your public ARPA/PTR record:
>nslookup
> set q=ptr
> 8.8.8.8
Non-authoritative answer:
8.8.8.8.in-addr.arpa name = google-public-dns-a.google.com
8.in-addr.arpa nameserver = ns1.Level3.net
8.in-addr.arpa nameserver = ns2.Level3.net
The host would then be set to google-public-dns-a.google.com.
The second worked, and I am not sure why for me on a project I worked on in the past, where you would have thought the IP address to be sufficient.
I am not sure if you have root access or access to my.cfg. If you can edit it, make sure the line "skip-networking" is commented or removed and it contains line "bind-address = *". Restart mysql after editing config.
For security reasons you shouldn't access the database directly over the (public) network.
One way is to write a php script on the database server and access it via HTTP/POST.
You should authenticate the client via username and a hashed password. The data you are sending should be encrypted (eg with the users clear text password). Don't send complete queries, only what you want to do and the parameters. As example, you want the orders for the customer, you can send a post request with the following parameters
user=abc,password=9151440965cf9c5e07f81eee6241c042a7b78e9bb2dd4f928a8f6da5e369cdffdd2b70c70663ee30d02115731d35f1ece5aad9b362aaa9850efa99e3d197212a,data=EncryptedData
You can notice, that the password is an SHA512 Hash.
The data can be json or anything else:
{
"Command": "GetOrder",
"Limit": "10"
}
In your php code you do the following steps:
1. Authenticate the user, if the password is not correct, respond with error code etc
2. Decrypt the data
3. Execute a query
4. Return the result as encrypted data
If you don't want to store the clear text password in your database, you could store in your database the hashed value and use a double hashed value for authentication and single hashed value for encryption.
If you wan't to execute the queries with parameters from the request you should use prepared statements to prevent sql injection.
More information about en/decrypting in php see: http://php.net/manual/de/ref.mcrypt.php
Like some answers suggested, I think you are firewalled by bigrock.
Now if you want to use AJAX/PHP, you need three things:
- your C# class to send requests and receive the result
- your HTML/JS (or jQuery) file to receive the request and hand it over to your PHP. Then send you the result.
- your PHP file to query your DB.
The AJAX seems superfluous to me, you could just send your query passing it through POST or a GET parameter (i.e. example.com/query.php?req='SELECT * FROM clients')
The code would be as follow:
C# using this class made by Ali Ahsan Rana:
//create the constructor with post type and few data
MyWebRequest myRequest = new MyWebRequest("http://www.example.com/query.php","POST","req=");
//use System.Web.Script.Serialization and myRequest.GetResponse();
Some tutorial on System.Web.Script.Serialization.
On the PHP side:
<?php
$request=$_POST['req'];
$dsn = 'mysql:dbname=mydb;host=example.com';
$user = 'ajay';
$password = '0000';
try {
$dbh = new PDO($dsn, $user, $password);
} catch (PDOException $e) {
echo 'Connection failed: ' . $e->getMessage();
}
$response = $dbh->query($request);
while ($rep = $response->fetch())
{
$response_array[] = $rep;
}
$response->closeCursor();
$return = json_encode($rep);
return($return);
?>
That's a quick draft but should work AFAIK.