I have a security course project. It asks to enter a given website and download its information 20 times(site has 20 subpages), then parse etc. I am using c#'s downloadstring to download and parse the page. However, after the fifth time, website finds out that I am doing those downloads as a robot(programmatically).
What I create as a program is successful until the sixth request.
I download the content and parse the desired information. When I reach the sixth subpage, my pc is blocked.
It is not related with time interval. Because, I used random generated timeouts between 6-12 seconds. However, that does not help. It is definitely related with entry counter of the webpage. It is like " not give permission after 5 request in 30 minutes. If it passes the limit then block it for a (or more) day". Since, I have been blocked for many times. I am using my phone's Hotspot.
I find a solution while I am searching on the internet. People are using IP changing methods via netsh etc. However, I think my IP is static (WiFi) and I could not figure out how to change it programmatically in C# Windows Forms App.
Because of that, I would like to hear your thoughts.
Your ISP most likely gives you a single Dynamic IP Address, which is the IP Address of your computer's access point to the Internet (i.e. the WAN). If so, they control the IP and not you. Even if you have a home network with multiple computers all on different local IP Addresses (LAN), you still aren't changing your WAN IP address which is the address that is effectively blocked.
Also, I am not going to judge, but I would say that if this is for an actual course project, then ethically speaking your instructor most likely would not want you to hammer an innocent website any more than the website's owner wishes for you to hammer it, hence the blocking. My suggestion would be to set your sites on another website that does not have the blocking to complete your coursework. Maybe you can do this against Google.com?
If you really need to make a request through a different IP address you could link your application up to several different proxies and switch between them at intervals.
Also, you mention that your IP is static, but there is a difference between your local IP and your external IP address. The IP address given to your WiFi connection is local, but the external IP address which is the one that would be seen by Internet sites is not the same.
If you have a dynamic external IP address one option might be able to programmatically connect to your router and restart it. This is one way to trigger an IP address update if you actually have access to it.
Overall, what you are doing is difficult to achieve for what sounds to be a simple assignment.
Here's a rather involved and eccentric solution that would, however, get around the problem nicely. Create 4 Amazon EC2 t2.micro instances (Windows) and issue 5 requests each from the EC2 instances. You can store the result to S3 buckets. It would take you a lot of work to get this working, but you'd come out the other end also having your first experience of working in the cloud. And each of those instances would have a different IP.
Also if you spin the same instance up and down a few times, it's unlikely to have the same ip in any case, so you could easily get away with one instance.
In a more serious vein: experiment with changing your user agent string and adding a much more hefty amount of time (minutes, hours) between requests. Also, turn your hotspot on and off between every five request, which will likely give you a new IP each time.
Related
There's a similar question ( Can I temporarily override DNS resolution within a .NET application? ) but I just can't figure it out with the meager response there.
Background info
I have a server set up in my home network, an old computer. Our router has the right ports forwarded, the server runs server software for things like http, svn, games, etc. I've got a domain name registered that always points to our external IP address. For all intents and purposes, I've got a typical webserver set up. My friends can game on my server by connecting via the domain name, I can push and pull svn projects, etc.
The only problem is that I also need to use my server when I'm connected to my home network (the same network as the server). Using the domain name results in Windows resolving it to our own IP address, and my router is too retarded to realize it just needs to forward it back into our network to the server as per the usual. I've done some looking around, configuring, telnetting and DNS overriding, but I have it on good authority that our ISP apparently crippled the DNS override feature of their routers to prevent this exact scenario. Apparently they don't like internal loopbacks.
I now basically have to keep 2 configs for each of my server's services: one config that specifies the domain name for when I'm abroad, and a second that specifies the server's internal IP for when I'm at home. It's frustrating because it just isn't always possible.
I want to instantiate a global DNS resolution request/response listener that will do the following: if the requested domain name matches a given string, override the IP in the response with one of my choice.
I've looked things up like easyhook, dllimport, msdn pages, etc, but I still can't figure out where to actually start, which classes I need to get access to, and so on. I basically have no pre-existing code for this particular problem.
I have Visual Studio, years of relatively simple programming experience and a good understanding of unfamiliar code and everything else, just no idea how to start or what to look for.
Many thanks for anything that can get me going.
Couple of pointers, DNS is as you mentioned some server somewhere that knows how to resolve a Domain name to an IP ADDRESS. There are two types of responses 1) Definitive for the domain and 2) Not Definitive for the Domain. Obviously any Definitive response will win.
DNS does not run at TCP/IP layer it runs one layer below using UDP (port 53). As a result DNS is tied into the TCP/IP stack whereby the stack first looks locally to a local host table and sees is the name is there, if it cannot find it the stack will send out the DNS request prior to the session start or SYN packet flying. In order for this to happen, the DNS servers must resolve the address.
I do not fully understand your issue because most Windows servers only have 1 IP Address and one associated Domain name. Unless of course they are using DHCP which just means the server uses multiple address on boot up and DNS server receive updates to what those new address are.
Public vs. Private IP addresses are the responsibility of a router. The router is configured to do NAT (Network Address Translation). This allows a private server to have unroutable addresses like 10.10.1.100 etc. But the router will answer ARPs for the public address making the public network think the router is the public address. So one of your solutions could be to convert your server to a private address and use a "real" router that can perform NAT. Port forwarding is not really routing rather it's a way to "trick" packets destined for one port number to be forwarded to another port dedicated for that application.
My suggestion is to either follow the suggestions above or add the Domain names you want to the local Host table on the server. This will get you what you want. A better solution is to configure your own server to be a DNS server definitive for your domain, that way you get to control everything.
I ended up solving this with (as a few have suggested) the hosts file after all.
I first used ManagedWifi to set a network connection monitor. It detects changes to my connection status and reports the network name.
Then I wrote a console app that stays open (using Hidden Start allows me to hide the window) and safely modifies the hosts file, then flushes the DNS cache. This seems to work in pretty much realtime. :)
I want to get the IP address I am being NATed behind (e.g. the one presented to the outside world, rather than my local IP).
Something like the result you get from www.ipchicken.com.
How can I get this? The local IP info is easy to find, but I have no idea how to go about getting the IP assigned from the ISP.
The reason I need it is that my network infrastructure is such that I have two gateways out of the network. If one fails, it trips over transparently to the other. All well and good, but there is no alerting mechanism to tell me that I have failed over.
I believe it's quite hard to get this information. I guess another approach I could take is by putting a webservice on an externally hosted webserver - the idea being that it returns the IP of the querying host. (However, I have no idea how to do this either!) I suspect this might be the easiest way to go.
Your computer does not have access to this information locally; you need to get it from one of the many network services which do this.
There are lot's of ip services
This one, for example. Just make a GET request and parce a simple html : http://checkip.dyndns.org/
Some more details about this service (Policies and rules):
http://dyn.com/support/developers/checkip-tool/
upd
If you need to check your IP frequently you'd better to add your own web-service. .dyndns.org allows to check the ip once for 10 minutes.
You can run some .net code on web-service:
((IPEndPoint)tcpClient.Client.RemoteEndPoint).Address.ToString();
But I think a script language (python?) will be more suitable.
How to build Opera unite like functionality using .NET?
When we run opera browser and enable the opera unite then it shares our desktop on web and we can access it over internet without having a static IP, how does this works?
You need to map your dynamic IP to some static DNS name. One easy to use solution would be a DynDNS service. For these you get a small client program (or good routers already have a dyn dns client already implemented), that simply maps your current ip address to a specfic dns name.
After getting this done it depends on how variable you like it:
- Do you have one master that should be able to look on various client PCs for help?
- Do you like to connect two clients which are both have dynamic ips?
In the first case your master is the one how always ensure to map his current ip address to the dns name. On the router maybe some port forwarding is needed and your clients get a vnc client with reverse mode enabled. This allows everyone to built up a connection to your master and sharing the desktop without any changes on the client site. Only on the master site has something to be done (dynDns and port forwarding).
In the second case it is a little bit more complicated. Now you got two dynamic IPs that like to connect to each other. This is normally be done through some kind of mediator. Means both machine connect to some master (reachable through some DNS name). Now both machines have a bi-directional connection to the master and the master will simply take the data from the first client and forwards it to the second and vice versa. To take all the burden from the master it would be possible that the master in a first step sends the ip address of client A to B and vice versa. Then in a first step both trying to connect to each other directly. This method is known as UDP hole punching and does not always work, which then leads to a fallback to use the master instead.
Two examples that are using the second approach are Skype and join.me.
All these techniques don't have to do anything with .Net. It is pure knowledge about ethernet, TCP/IP, internet, etc. And due to the fact that you have to do a lot of work in this low-level area i would think that C# is not the ideal framework to accomplish this task. Maybe to put the GUI as last step over it all, but for the real work behind the scenes you need some decent network know-how which is mostly independent from the programming language you are using.
You should run your own DNS server. And whenever an user opens the browser and logs in, you should send the current IP address of the user to your server and update in your local DNS. Since you will have only a subdomain (desktop.operaunitecom), whenever someone hit your url it first goes to operaunite DNS server from where it will fetch your IP address. When the user's session closes you should remove the DNS entry. This could be a possible solution.
I have 1 computer with 1 network card installed. The network card has 10 IP addresses assigned to it. I have a windows desktop application running there. The application is basically a webbrowser making a call to 1 specific website.
What I want to achieve is for the webbrowser to change its source IP address (round-robin through the IP addresses available on the network card) every time I visit the website, so that the website sees 10 different external IP addresses with each visit.
To add to this, id like to have a scenario were two web browsers run concurrently through different ip addresses
I'm looking to do the application in C#. How do I do it? Or perhaps can anybody point me in the right direction?
There's a lot to this question. If you want the other website to see your address then you need to ensure there is no proxy because proxies by their nature hide your address. Ensure all 10 addresses are public. NAT isn't going to expose 10 different addresses either.
Assuming you are using Berkely sockets, you can loop through the logic to create the sockets and flip out the endpoints in whatever cycle you desire.
while(true)
{
EndPoint ourEP = new IPEndPoint(MyServerIPOne, serverSocketPort);
socket.Bind(MyServerIPOne);
socket.SendTo...etc
}
Example on MSDN displaying how to connect via http.
I have a list of IP addresses of bots/hackers that are constantly attacking one of my sites. I want to block these visitors by IP and am trying to work out a "best" approach for this. My site uses C# ASP.NET MVC.
I have a List<int> of IP's.
Where is the best place to put the check code? I'm thinking of using the Page_Load event of a master page but could also put it in a filter to each controller...
What HTML do you return to the banned IP? I am reluctant to return a "site blocked because your IP is banned" because this will give the hackers the information they need to work around the block. The advantage of doing this is that it will give the innocent users who have been caught in the crossfire the reason why they can't access the site. My current feeling is that I should return a "Site under maintenance" notice.
What HTTP status code should I return with a fake "Site under maintenance" notice? I'm thinking 200.
Site is running on Server 2003.
If you feel your site is being "hacked" from a specific IP, you should not be blocking that IP in software, the very thing that they intend to compromise. Blocked IPs should be blocked at the firewall.
I'd have to agree with David on this for several reasons.
By blocking via software hackers/bots will still be able to abuse your resources (bandwidth, processor time, etc).
Software cant protect your site against dos attacks.
If a hacker is good they'll find a way around software blocks.
Updating blocking code will require recompiling of your application.
Your answer is in the firewall. Set up rules to block out the users and they wont be able to connect.
Sending an "under maintenance" page is a terrible idea because it'll confuse normal users and won't deter a good hacker...
While you could block the IP addresses on your outward facing servers (your web servers obviously but you may have others) this list will need to be replicated across all. By blocking on a server you're not only overcomplicating the solution but also providing a method which is not wholly secure.
The proper point to block network traffic, whether it be a select list of ports or IP addresses, is as far out on your network as you can get. This is typically a firewall/router at your entry point. These networking devices are optimized for this very purpose, as well as far beyond that. Depending on the manufacturer of your networking equipment the feature set will widely vary.
I suggest you:
Identify all routers/firewalls at the
outermost boundary. It is possible
you only have one unless you're load
balancing.
Learn how to configure the ACL
(access control list) for those
devices.
Modify the ACL based on your IP
addresses list to block.
Always save a backup of your network
device config elsewhere.
Obviuosly this is just the tip of the iceberg in security. Perhaps at some point you'll need to contend with DOS (Denial of Service attacks) and then some - oh the fun.
Good luck.
I'd stick the code in a place where it will run as soon as possible, before the server consumes too many resources .
I would say you should send back as little information as possible, ideally HTTP status 503 (Temporarily unavailable) with a short message linking to an acceptable-use page, or a page explaining to people some reasons why they MIGHT have been blocked and what to do if they feel them are blocked unfairly. You may wish to do this in text/plain instead of HTML as it will use fewer bytes :)
Using an in-memory list of blocked IPs also breaks when you have a large number of blocked addresses (say 1 million) because scanning it becomes prohibitive (remember you need to do this for every request to the relevant resource).
Ultimately you will want a way to distribute the lists of blocked IPs to all your web servers and/or keep it centralised - depending on exactly what kind of abuse you are getting or anticipating.
Having said that, you should definitely apply the YAGNI principle. If you aren't experiencing real capacity problems, don't bother blocking abusers at all. Very few sites actually do this, and most of them are things where there is a significant cost associated with running the site (such as Google search)