Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I'm trying to bootstrap a micro ISV on my nights and weekends. I have an application at a very early stage of development. It is written in C# and consists mainly of a collection of classes representing the problem domain. At this point there's no UI or data persistence. (I haven't even settled on the .NET platform. Its early enough that I could change to Java or native executables)
My goal for this application is that it will be a hybrid single user/ occasionally connected multiuser application. The single user part will use an embedded database for local storage. This is a development model I'm familiar with.
The multiuser part is where I have no prior experience. I know each user will need two things:
IP based communication to a remote server on the public internet
User authentication and remote data storage
I have an idea of what services I want this server to provide (information lookup and user to user transactions) but beyond that I'm out of my element. The server will need to be hosted by a third party since I don't have resources to run my own server. Keeping in mind that I will be the sole developer for this project for the foreseeable future:
Which technologies would be the simplest way to implement the two things mentioned above? Direct access to the datastore/database or is it better to isolate it? Should I implement a webservice? If so, SOAP or REST?
What other things do I need to consider when moving to a multiuser application?
I know security is a greater concern in a multiuser application. Especially when your dealing with any kind of banking information(which I will). Performance can be an issue when dealing with a remote connection and large numbers of users. Anything else I'm overlooking?
Regarding moving to a multiuser application, centralising your data is the first step of course, and the simplest way to achieve it is often to use a cloud-based database, such as Amazon SimpleDB or MS Azure. You typically get an access key and a long 'secret' for authentication.
If your data isn't highly relational, you might want to consider Amazon SimpleDB. There are SDKs for most languages, which allow simple code to store/retrieve data in your SimpleDB database using a key and secret, anywhere in the world. You pay for the service based on your data storage and volume of traffic, so it has a very low barrier of entry, especially during development. It will also scale from a tiny home application up to something of the size of amazon.com.
If you do choose to implement your own database server, you should remember two key things:
Ensure no session state exists, i.e. the client makes a call to your web service, some action occurs, and the server forgets about that client (apart from any changed data in the database of course). Similarly the client should not be holding any data locally that could change as a result of interaction from another user. Cache locally only data you know won't change (or that you don't care if it changes).
For a web service, each call will typically be handled on its own thread, and so you need to ensure that access to the database from multiple threads is safe. If you use the standard .NET or Java ways of talking to a SQL database, this should be handled for you. However, if you implement your own data storage, it would be something you'd need to worry about.
Regarding the question of REST/SOAP etc., a key consideration should be what kinds of platforms/devices you want to use to connect to the database server. For example if you were implementing your server in .NET you might consider WCF for implementing your web services. However that might introduce difficulties if you later want to use non-.NET clients. SOAP is a mature technology for web services, but quite onerous to implement, and libraries to wrap up the handling of SOAP calls may not necessarily be available for a given client platform. REST is simple to implement (trivially easy if you use ASP.NET MVC on your server), accessible by any client that can handle HTTP POST/GET without the need for libraries, and easy to test, so REST would be my technology of choice.
If you are sticking with .net (my personal preference), I would expose data access calls via WCF. WCF configuration is really flexible and pretty easy to pick up and you'll want to hide your DB behind a service layer.
1.Direct access to db is the simplest, and the worst. Just think about how you'd auth the db access... I would just write a remote-able API with serializable parameters, and worry about which methods to connect later (web services, IIOP, whatever) - the communication details are all wrapped and hidden anyway.
2.none
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
Suppose I have an API endpoint such as Facebook Graph API, which I design an application running on my PC to periodically connect to the API and retrieve my posts, comments, etc. On each Timer_Tick, the program reconnects to the API and brings the top 10 data items from the API, and persists these data into databases.
Now, suppose that this application is built by 3rd party, and I just downloaded from the internet as binary file not opensource.
How can I know if the application is leaking my Facebook data to third party without my knowledge?
Is there a mechanism to monitor such leaking if found? (from programmatic perspective)
This is a matter of security and for being sure you almost must think about any vulnerability here and try to make sure there is no way to reveal data by the known vulnerabilities but you cannot be sure about unknown ones.
If this is the matter of the trust and you are dealing with sensitive data i strongly recommend you to avoid using 3rd party tools unless they are provided or certified by the API provider. here are some techniques witch will help you understand about what is going on in the backyard but they will definitely not guaranty the safety :
1- First of all make sure the application is really a binary code (i know you mentioned it as a binary), it's because some executable files are just scripts or semi-scripts but look a like a binary files. for instance in the some cases if the source of the executable application is written with C#, Python, Java, there are tools out there that will help you DeComplie the application and find out what's going inside. this solution of course can be considerably tough if for example the code is obfuscated or there is complex models or OO programming models involved.
2- Use network monitoring tools like WireShark or any other tool to capture all traffic of HTTP/HTTPS requests while using the 3rd party application. because the API is just the same as HTTP requests used by applications to exchange data you can use these tools to monitor what's going on in your computer. normally this application must only connect to the Facebook servers and URLs needed to use the web API, if there is any other request sent or received from a server other than the Facebook there is chance of data leak here. if these requests are not encrypted by SSL/TLS you would be able to see the data being exchanged or if they are encrypted through SSL/TLS there are tools that provide man in the middle attack solution to see these traffics but if they are encrypted in the application layer you won't be able to see what data are being transmitted so it might involve suspicion about data even higher chance of data leak. don't forget that this monitoring must be extended for the entire using cycle of the application.
Also limiting the application to talk only to the server in witch you are calling the API with OS Firewall will be step forward to decrease the chance of data leak here.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have to write an api-client system that connects to multiple api-servers, does a job and disconnects. It does two simple things, but needs to do it at scale (ie: aiming for 200-500m outbound API client calls per day):
(1) Simple client connects to an API-server (http/rest), sends a query, receives a response (text based), saves the response for later, and moves on to the next server/query.
Once responses start coming in, a separate process will:
(2) parse the text in the responses and add them to a large file/queue for reporting
I currently have a test system in C#, running 20 console applications on a machine, with 20 threaded clients in each console application carrying out the work. I need to be able to scale this up on demand. What is the best approach to do this? ... I am sure a solid pattern exists to this simple problem?
My thoughts so far are:
-> design a management system that depending on the volume of API-servers to be queried in a given hour, orchestrates the provisioning of virtual machines (not trying to redesign the wheel - will hook into any existing framework like chef/puppet etc if suitable)
-> have a central system for collection of data from the api-clients (perhaps a node instance passing the data off to RabbitMQ for later pickup/processing)
-> have a separate management system that orchestrates the text parsing of data received from the API clients.
-> As project is network latency bound, I believe development language is not really relevant so long as it has good network support.
My main questions then are around:
(1) What would be a most appropriate language/framework to implement this in to enable a lean/cost-effective system? ... ie: no point in spinning up multiple Windows VMs for example if they have a bigger footprint/overhead/cost than doing the same thing in linux? (so in this case I could use the mono framework - get the benefit of C# that my team knows, but the lower cost of linux VMs...)
(2) Is my thinking about having to spin multiple VMs up to do this correct (albeit small VMs running X client applications each)?
(3) Another approach I thought of is to write the clients in Javascript - the reason being that the bottleneck for the api-client is network and api-server response time, not client-side, so it might be well suited to async work? .... in this case I could have one Node server running 100x more api-clients than I could ever get in even a bunch of micro-windows VMs ?
(4) Finally, am I reinventing the wheel? ... is there anything out there on Amazon or Azure already that I can plug into that would provide a ready framework for what I need?
All comments and suggestions and guidance most welcome.
Many thanks.
I am not a specialist in what Amazon provides. Here is what you can use on Azure depending on your needs:
Worker role - this is pretty much a scalable virtual machine. You can scale out or autoscale by condition.
AppFabric and Microservices - for more complex deployment and more granulated development infrastructures.
Azure Functions - an interesting scalable and cost effective processing option. Check it out.
In terms of choosing the language, I would use Node.js if your application is not too complex and it's not going to in the near future. C# is better for more solid systems with complex architecture. Both platforms are supported on Azure.
Have a central system for collection of data from the api-clients
(perhaps a node instance passing the data off to RabbitMQ for later
pickup/processing)
If you need a really big throughput, RabbitMQ may not be enough. On Azure you can use EventHub. More info here.
"Finally, am I reinventing the wheel?" Its a good question - you might be. From your description, you have a lot of proprietary management of servers going on - and a lot of VMs. Depending on your workload, you may not need need manage any traditional VMs at all. Avoid that if you can to keep things lean. There are some great technologies that make server management (patching, security, server administration, etc) a thing of the past for many work loads: event-driven computing frameworks such as AWS Lambda.
Consider a server-less implementation using the API gateway pattern, and microservice architecure pattern, using the following AWS services:
AWS Lambda is a compute service where you can upload your code to AWS Lambda and the service can run the code on your behalf using AWS infrastructure. After you upload your code and create what we call a Lambda function, AWS Lambda takes care of provisioning and managing the servers that you use to run the code. Very light weight. The first 1 million requests per month are free
"Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale." $3.50 per million calls. Scaling, security and management all built in. Lambda supports the specification of HTTP endpoints via the API Gateway to trigger Lambda functions.
AWS Lambda provides an easy way to build back ends without managing
servers. API Gateway and Lambda together can be powerful to create and
deploy serverless Web applications. In this walkthrough, you learn how
to create Lambda functions and build an API Gateway API to enable a
Web client to call the Lambda functions synchronously.
You can also integrate DataPipeline for data transformation, and Simple Queueing Service for queuing/messaging, if needed you your workloads.
If you're doing anything stateful and at scale Service Fabric might be the better choice over Azure Functions/Lambda or Worker Roles.
https://azure.microsoft.com/en-us/services/service-fabric/
I'm about to release a small tool which uses a database connection for storing data. The question is: How can I prevent people reverse engineering my code and getting the Username and Password to gain access to the database?
For earlier projects (which were used only by myself), I defined the connection-string just as a global variable inside my app. But that's highly unsafe as it only takes minutes to get this string out of the exe.
Also a lot of methods to obfuscate code can be reversed.
I am really a big fan of providing code but I don't know what to post. This is more a question about the theory. Coding is the part I'll take care of myself.
Here is a small idea from me which I don't really like that much:
I could place a second tool on the server. The real app would connect to this second tool, give over the data and the second data would finally connect to my database itself. This way the connection-string would be stored inside the second app where nobody can grab it.
The fact of the matter is that storing sensitive information on the client machine is highly vulnerable to attacks against your database. A suggestion you can look into is a Three-tier architecture model for your application (http://en.wikipedia.org/wiki/Multitier_architecture#Three-tier_architecture). In a Three-tier architecture, you have your presentation layer (your application), your logic tier (this layer will be the central pit stop for all your clients will have access to your database), and you have your database layer (the server where your database is). With this architecture, you can ensure all the data being stored and being retrieved from is from a singular source and high level security.
In the past (and still in the present), programmers would have to create their own socket servers or do advance network programming to develop a solution like this, however Microsoft has developed a tool called Windows Communication Foundation (WCF) which takes away the pain of coding your own socket server and lets you focus on developing your own implementation. Be warned though, WCF is secure by default, but it is no excuse not to research into ways of making your product robust against hackers (like knowing what protocol you are going to use, what security measures you are going to use (Transport vs Message, etc), encrypting data on client side so potential viruses don't uncover sensitive informations, etc). In saying that, WCF is a highly polished service and is really easy to get something up and running.
A good beginner video tutorial on WCF can be found here: https://www.youtube.com/playlist?list=PLhq7kqloVlM-bI9W_7iDZhObAeyrFt1y_
EDIT: The playlist for the videos are gone, but the videos themselves are still there. Just search through all his videos looking for the keyword 'WCF'
Here's the link: https://www.youtube.com/user/JesseDietrichson/featured
Is it wise to build a large application entirely based off SOA? Or just some portions? User account logins, accounting, gis mapping, sales, etc?
In other words, would it be wise to build a GUI to such an application in HTML & Javascript which does all it's exchanges via ajax to .NET web services on the back-end?
I can't see it worth loosing all the .net .aspx functionality such as forms authentication, view state, etc. But my co-worker is saying if we are going to go SOA there is no need for .NET on the front end. But i think there should be some sort of balance. Where do you draw the line? Should all calls to the database go through the web services?
I just want to say that "with SOA we’re building for change, while with Traditional systems engineering, we’re building for stability."
The problem with stability, of course, is, it only takes the business so far — if the organization requires business agility, then they’re much better off implementing SOA.
So, It solely depends on what you want to achieve, you are the one who should draw the boundary.
I read it in article on SOA few days back as I'm too working on SOA.
EDIT:
Meanwhile I came across this article and thought of sharing with you.
The video quite explains the current scenario of SOA and its views by different people.
I'm getting the words of the song 'If I had a hammer' coming to mind. SOA is an architectural approach to develop software as a series of services. In my opinion this is best for systems that have less than immediate latency and limited bandwidth, and high cost in access etc (these are all obviously highly subjective). You don't need full SOA just get loose couping between components which I would argue is a good goal to achieve.
DB calls can go through a service, take ADO.NET data services for example however you really have to weigh up with what the service is to provide. Take caching. A decent approach to SOA will consider that data is may need to be cached to reduce service load. So can your data be stale in the UI? Are you allowing that use case? Is right for login info to be stale (a rough example I know but possibly something that may need to be addressed).
All in all - it depends. I think some things lend themselves to SOA very well. If you take a DDD approach then the services that represent Domains would probably do so. In this way your UI talks to domain services and not rows in table as the DB is abstracted behind domain services.
Don't use one methodology to solve all problems.
See this SO question too
It's a service oriented architecture, not a service exclusive architecture.
Presentation logic and plumbing have to live somewhere; it all depends on where it makes the most sense for it to live.
For example, let's say you have a UI component that relies on a highly chatty but efficient set of calls to a database to generate a complex analysis of something (take your pick). If your web browser is making all those calls, you introduce massive network latency and concurrency issues. If a web service makes all those calls, you are potentially putting presentation logic into it to format that result.
If you are using Session state (or web services period), you are essentially using ASP.Net anyway. Try uninstalling it and see if your web services still run.
If presentation logic needs to live on the server side, it is better for it to live within a framework intended for presentation rather than a web service, IMO. If you haven't looked at MVC 2, do so. It makes it incredibly easy to set up an application that melds browser and server UI support (for example, jQuery validator controls backed by server-side validation).
Conversely, the web browser provides an expressive platform. Assuming browser support and team knowledge, the AJAX/SOA architecture you describe is a good one. I'm using it more and more and trying to make my server pages cleaner and simpler but I have no plans to exclude ASP.Net from my toolkit any time soon.
Client implementation should be completely disconnected from the back end web service in a SOA. The service should be able to be consumed by ANY client. If you are using .NET on the back end and front end because they can be coded to directly communicate, then you are missing the point, because now they are tightly coupled and what you have now is a stove pipe application. The client should have no idea how the server side is implemented -- shouldn't matter if the back-end web service is built using .NET, Java, or whatever.
In a true SOA, you should be able to search for services in the services repository, perhaps tie the outputs in with other services or use XSLT to create alternative outputs that weren't necessarily considered when the original service was built, and consume it in a standard way in any client on the front end.
It sounds like what you're really asking is how to build a single application. The point of a SOA is to provide standard data sets through re-usable interfaces, that have no specific application or implementation in mind. To start out building a single application with the entire back-end comprised of SOA services would be a huge undertaking. In MY mind, each back-end service should be built because of it's intrinsic value all on it's own and be provided to the entire SOA "domain". Then when you or I decide to make a client that does X, Y, and Z, we can just go find those capabilities in the SOA and injest them.
Note: When I refer to tier, I mean a physical tier. Many of the questions on this site relating to "tiers" are referring to logical layers, which is not what I'm asking about.
I am designing an app using a standard "3 layer" architecture, with presentation, business logic (BLL) and data access (DAL) layers. The technology is WPF, C#, LINQ and SQL Server 2008. My question relates to the physical architecture of this app.
I can place the BLL/DAL in a standard DLL which is loaded and run on the user machine, making a 2 tier architecture - client machine and database server. But it is not too difficult to turn the BLL/DAL into a WCF service which sits on an app server and is called from the user machine. This would give me a 3 tier architecture - client machine, app server and database server.
My question is this - what is the advantage of using a 3 tier architecture? I've often been told that 3 tiers add scalability, but it's not immediately apparent to me why this would be. And surely you are going to take a performance hit with the same data having to make two hops over the wire - from database server to app server, then from app server to client machine.
I would appreciate the advice of experienced architects and developers out there.
It depends on the use of your application and your requirement for security. If your application is being used over the Internet, and you're storing anything that is potentially sensitive in any way, adding the physical remove for the database is strongly recommended. Never, ever let anyone from the outside onto any machine with direct access to your database. People can and will attempt to break your security for no better reason than they have nothing better to do.
Scalability can be a factor as well, both in front of the presentation layer (in front of the web servers) and in the database. Placing a load balancer in front of the presentation layer allows incoming requests to be routed to an array of machines that can be managed independently. Machines can be added to the pool in times of need and removed for maintenance. Placing load balancers between the other layers can have the same impact. The idea is to provide a flexible, dynamic back-end environment that can be adjusted as demand requires.
Whenever you find yourself asking the question "Is X inefficient?" you should, immediately, ask yourself three precursor questions:
By "inefficient," what resource do you think it should be using efficiently and may not be? Time? Space? Bandwidth? Development hours?
Why do you care? No, seriously: If you're going to spend even one minute answering this question, there has to be a reason. What is that reason?
Compared to what?
As far as your comment about scalability is concerned: For a time, I had the unfortunate responsibility of maintaining a system whose architect who had been told that minimizing round-trips to the database would make an application scalable. He took that insight and ran with it. You can read about this project here. It occurs to me that I ought to have mentioned that at no point during the entire decade-plus-long lifetime of that application were there more than four users logged in simultaneously.
Inefficiency is in the eye of the beholder.
For example, having everything happening on the client may increase the memory footprint or CPU/network requirements of the client computers. If this work can be off-loaded to a server/server farm you may save having to do hardware upgrades of client PCs just to run your software. If more resources or upgrades are needed, they can be added/done in the business logic tier without impacting the clients.
Also, having the business logic on its own tier may be more efficient later (from a software development perspective) when you need to expose some of your application's functionality in a web-based system, or an Outlook add-in, or an iPhone app. You don't want to have to update all of these systems whenever the business logic changes slightly.
Security may be better as your users don't need direct access to the database server, they are isolated by the application server.
It also forces you to think about your application in a modular way with well defined interfaces which may have architectural benefits to the design of your application.
It can be. It depends on what has been implemented and how.
The driving force for creating a 3 tier physical architecture is not necessarily performance related.
The reason scalability is quoted is that a service might run on a server farm, but the clients would be unaware of this. The size of the server farm can be increased to meet demand if the architecture has been designed to support it.
Main advantage of 3t applications described like you did is not scalability. Maintainability maybe.
In order to make your architecture scalable you need one more technology you didn't mentioned.
- you need services. I would suggest WCF.
Making your BLL WCF service (or multiple services) would make your application much more efficient and scalable, allowing your BLL to run on different/multiple machines.