There are few more things you can do other than adding hardware and caching…
- combine small files. A lot of small files is slower to download than a few large files
- move static resource files such as images, CSS, and JS files off of the ASP.NET servers, and set cache settings on that server. A server tuned for ASP.NET is not especially well-suited to server these files. From the client side, the browser can open two additional concurrent connections to download the static files. Another benefit is that the browser doesn’t need to send the cookies to the static files (ASP.NET cookies will be sent to web apps otherwise).
- think twice before breaking services into different server. All those out-of-process calls between the web servers and business object servers create a lot of overhead.
- IP affinity might be easier at the beginning (so that session can be managed in-process), but will create grief in the long run/bigger load. Scenarios: IIS process recyles, some megaproxy (ie AOL) will be bound to a single server etc
- use compression. Yes it costs processor cycles but you typically have plenty of extra CPU capacity on a dedicated web server. IIS7 is even optimized so that when the processor gets really busy, it will suspend compression. Or check http://www.port80software.com/ for 3rd party solutions
- optimizing the HTTP pipeline. There are some HTTP Modules that sit in the ASP.NET request pipeline by default that you may not need. For example, if you don’t need session management, and only use FormAuthentication, you can remove Session, WindowsAuthentication, PassportAuthentication and FileAuthorization in httpmodules section.
- reduce or even completely disable the ViewState
- GET is preferred over POST for AJAX calls. GET carries much smaller footprint than POST and POST costs one extra HTTP call than GET. However POST has security advantages so leave it to POST if that’s a delete/update transaction or involved sensitive data.
- programmatic caching add complexity but has great benefit too
- scaling database. Move to cluster, partition the tables, use dedicated reader and writer databases (and of course the sync process too)