Skip to main content

Questions tagged [scaling]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
105 votes
8 answers
221k views

What limits the maximum number of connections on a Linux server?

What kernel parameter or other settings control the maximum number of TCP sockets that can be open on a Linux server? What are the tradeoffs of allowing more connections? I noticed while load testing ...
Ben Williams's user avatar
  • 2,396
48 votes
5 answers
99k views

How many requests should my webserver be able to handle?

Not going into specifics on the specs since I know there is no real answer for this. But I've been doing load testing today with the ab command in apache. And got to the number of 70 requests per ...
Ólafur Waage's user avatar
22 votes
2 answers
18k views

How can I make Microsoft Management Console high-DPI aware in Windows 8.1?

Since Windows 8.1 doesn't allow system-wide "Windows XP style" high DPI support, how can I make the Microsoft Management Console apps (mmc.exe) high-DPI aware? There is no "Troubleshoot compatibility" ...
user20939's user avatar
  • 385
20 votes
4 answers
10k views

Which database server scales better: PostgreSQL or MySQL?

I'm working on a website that could eventually have a very large database. The website will be an art community not until deviantART. I'll be running a lot of queries for analyzing how the site is ...
epochwolf's user avatar
  • 639
19 votes
4 answers
19k views

How to allow elastic load balancer through port 80 in security groups?

I want to temporarily block port 80 for the outside world, but want the load balancer to pass through the firewall (via security group) so it won't see the instance as unhealthy. How can I do that? ...
Liron Harel's user avatar
17 votes
3 answers
9k views

Is there a way to add more backend server to haproxy without restarting haproxy?

We want to be able to add more backend servers on demand. Right now I don't see a way to add more backend servers to the config file without restarting haproxy.
Jan Deinhard's user avatar
  • 2,403
16 votes
2 answers
19k views

Tips for maximizing Nginx requests/sec?

I'm building an analytics package, and project requirements state that I need to support 1 billion hits per day. Yep, "billion". In other words, no less than 12,000 hits per second sustained, and ...
linkedlinked's user avatar
13 votes
5 answers
842 views

How do services with large ingest rates install enough hard disks fast enough?

Amazon, Facebook and Google must take in TBs if not PBs per day, so does that mean they have people continuously installing new hard disks and cables in new racks in the same way you might build a ...
jl6's user avatar
  • 2,635
11 votes
5 answers
14k views

AWS: Multi-region setup using single RDS instance

I'm trying to scale our web application (PHP, MySQL, memcache) in a multi-region scheme. Currently we are using a setup with two EC2 instances behind an ELB and an RDS instance, all of them in US-EAST ...
Ion's user avatar
  • 278
10 votes
6 answers
14k views

How to identify heavy write to disk?

I have this problem with server running CakePHP application. The server is insanely slow, I first thought that it's application problem, but then I found constant 5-6MB/s write to disk. What is the ...
Jakub Arnold's user avatar
  • 1,804
10 votes
1 answer
7k views

Recommendations for Elastic Search hardware [closed]

Are there any good guides for hardware level to support ElasticSearch? Are recommendations for Lucene or Solr a good place to start? We're looking at rolling out a deployment starting with 27 million ...
James Socol's user avatar
9 votes
8 answers
1k views

More RAM vs. more servers

I was recently asked "Do you know when to decide between going for more RAM or more servers?" (in the context of scaling data mining applications). I had no idea, so what are some ways to decide? I ...
user avatar
9 votes
3 answers
9k views

Postgres vs MySQL: Which one is more difficult to scale out?

From your experience, which database serve is more difficult to scale? MySQL or Postgres? I know MySQL has several out-of-the-box scaling/clustering features, but with Postgres there is none that ...
Joshua Partogi's user avatar
9 votes
2 answers
2k views

Large VPN network (~600 servers) with OpenVPN

I'm doing a preliminary study for a contract to build a VPN network between ~600 remote servers running Linux CentOS 6 (+ their 600 private LANs). The network is supposed to be star-based, so that ...
Giovanni Bajo's user avatar
9 votes
3 answers
2k views

The best way to share a file uploads directory across horizontally scaling web servers

I'm currently trying to spec up a horizontally scalable cluster for a drupal based web app, that looks something like the colorful diagram below: The load balancer implements sticky sessions, so a ...
Chris Adams's user avatar
8 votes
10 answers
26k views

Where to purchase a server capable of 512GB of RAM? [closed]

Where can I purchase a single server capable of 512 GB, yes GB, of RAM? The most I've seen is 256 GB (32 DIMM x 8 GB per stick)
user avatar
7 votes
4 answers
7k views

Load balanced Drupal servers: a good way to share data among the servers?

I'm trying to get an overview of how i would go about creating a load balanced web server setup. Setting up the actual load balancer and adding two or more web servers seems fair enough. However, I ...
sbrattla's user avatar
  • 1,630
7 votes
1 answer
2k views

How do I scale my EC2 instance?

I'm very new to server configurations and Linux in general, so please bear with me: I have a small EC2 instance running a LAMP server on Ubuntu 10.04 LTS. I'm completely baffled as to how I would ...
Calvin's user avatar
  • 413
7 votes
1 answer
10k views

How to horizontally scale SSL termination behind HAProxy load balancing?

I've been looking around and no one seems to be trying to scale SSL termination the way I am and I'm curious as to why my approach seems so uncommon. Here's what I want to do followed by why: 10.0....
Magna's user avatar
  • 181
7 votes
3 answers
702 views

how to go about scaling a web-application?

I am primarily a web-application developer, and I do not know much about scaling/scalability techniques. My application is written in Python, using Django; a fairly standard setup. I currently use ...
phoenix24's user avatar
  • 179
7 votes
2 answers
4k views

HAProxy not balancing server load fairly

HAProxy doesn't seem to keep the connections to the servers balanced. Keep this in mind: using HAProxy v1.3.26 5 equally balanced server specs algorithm is round robin, but no weights are applied ...
Mark's user avatar
  • 71
6 votes
2 answers
1k views

AWS - how to serve faster to oversea users

We have our infras relying on AWS in the us-east-1 region. (EC2, CloudFront, RDS, ElastiCache) We are now having more and more users from the APAC. Users start to complain about network speed to our ...
Chris Lam's user avatar
  • 283
5 votes
6 answers
5k views

How can I have 2 database servers for my website?

I have a site that has heavy traffic and my database server (mysql) is getting heavy traffic at peek hours. Instead of upgrading to a better server, I am thinking of having 2 database servers working ...
mr didodido's user avatar
5 votes
5 answers
4k views

When is the right time to add a separate database server, add more web servers?

Usually web projects start small, everything is on the one server. But if site becomes popular one server won't be enough. So, since I don't have any experience in this, I'd like to get an idea of: 1. ...
z-boss's user avatar
  • 301
5 votes
1 answer
590 views

Sharding MongoDB internationally with data stored based on location

We're looking to set up data centers globally and switch to a NoSQL database like MongoDB to support having our data distributed over these data centers (and as a pleasant side effect, introduce more ...
tzar's user avatar
  • 153
5 votes
2 answers
1k views

Memcached scaling strategy

Currently I am running a production environment with 4 dedicated memcached servers, each of them having 48Gb of RAM (42 dedicated to memcache). Right now they are doing fine, but traffic and content ...
danakim's user avatar
  • 402
4 votes
3 answers
6k views

Suggestions on large scale web applications architecture?

It's a big question:) We are running a website with LAMP that is not big, 5 web servers with LVS load balancing, 3 MySQL servers with replications and separation of reading and writing, and we use ...
Mickey Shine's user avatar
4 votes
4 answers
19k views

Maximizing TCP connections on HAProxy load balancer

I am currently using HAProxy in order to load balance tcp connections from clients to my Erlang app server. The connection is persistent, which means I'm limited to roughly 64K clients on an optimized ...
imaginative's user avatar
  • 1,991
4 votes
4 answers
777 views

Multiple web servers sharing a db server

I am contemplating migrating to mosso to have several cheap app servers. The goal is to have a dedicated database server, however, does it make sense to have one database server that hosts the ...
phillc's user avatar
  • 141
4 votes
5 answers
2k views

HAProxy and "sharding"

I am cross posting this from stack overflow since it was suggested I might get a better answer here ... I was wondering if anyone has used HAProxy for sharding. Specifically, I would like to be able ...
user avatar
4 votes
1 answer
2k views

Creating a stable, auto-scaling app server cluster

I have two servers, each running anywhere from 50-100 different legacy webapps, written in languages ranging from PHP to Python to Ruby-on-rails to NodeJS. We're wanting to kill off these machines ...
aendra's user avatar
  • 177
4 votes
1 answer
3k views

Can I use MongoDB with Amazon EC2 small instance?

I want to move my MongoDB from a virtual server to amazon web services. I know MongoDB provides a preconfigured AMI, with three provisioned EBS volumes. But this is only available for m1.large ...
Emii Khaos's user avatar
4 votes
3 answers
217 views

First steps in scaling servers up for a Ruby on Rails application

I'm running a RoR web application on a dual core AMD Athlon 4450e server w/ 8GB RAM and Ubuntu. It's doing alright, getting spikes up to 320 dynamic requests per minute, it's also handling all the ...
Tim Harding's user avatar
3 votes
11 answers
5k views

Active Directory for web authentication: scale to 1M users?

I'm interested in how well Active Directory would fare as the authentication backend for a website, scaled for ~1 million users. Do you have experience with AD in web environments of this scale, and ...
Parand's user avatar
  • 748
3 votes
4 answers
2k views

How to choose size for a cloud server (rackspace)

We're going to test the rackspace cloud next week to see how it's working with our web app. It's a LAMP environment with a lot of MySQL databases. How do I choose the "right" server size? On ...
Emil's user avatar
  • 133
3 votes
1 answer
462 views

Firewall as a guest VM?

Out of curiosity, would it be pointless/wasteful/silly to put a firewall as a VM guest (regardless of VM host - ESX, Xen, Hyper-V, etc. etc.) and redirect all traffic from other VM guests through the ...
osij2is's user avatar
  • 3,875
3 votes
2 answers
270 views

Virtualization as a way for scaling

I'm asking this question to find out what are you best practices to prepare your system for scaling, I know that basically you don't have to over optimized before really having to, and that's the best ...
Adam Benayoun's user avatar
3 votes
2 answers
3k views

horizontal scaling and load-balancing for PostgreSQL with ProxySQL

I have a PostgreSQL DB running in Google Cloud SQL that I need to scale horizontally. Cloud SQL makes it easy to add read-only replicas, but does nothing to help with load-balancing traffic to the ...
Toby 1 Kenobi's user avatar
3 votes
2 answers
304 views

RT database scaling

Recently I heard someone suggest that RT request tracker may have scalability issues due to its non-normalized database (someone at a Perl meeting I went to referred to it in a positive light as hyper-...
rplevy's user avatar
  • 131
2 votes
7 answers
6k views

How to deploy web application for 2 million users?

I've been tasked with figuring out how to do a large scale deployment for a high availability, high traffic, dynamic, flash based web application. There is a good possibility that this application can ...
lysdexic's user avatar
  • 366
2 votes
2 answers
3k views

Why is my AWS instances suddenly becoming irresponsive reporting high "stolen" CPU

The setup I have a bunch of t2.small EC2 instances running hosting the image processing library called thumbor for simple on-the-fly image resizing. Originals are loaded from S3. In front of the ...
Niels Kristian's user avatar
2 votes
3 answers
1k views

How do multiple servers work in terms of databases? [closed]

I don't know an awful lot about servers, but I do know that there are many ways you can improve the performance of running a web application, in terms of hardware. For example, starting with a single ...
Alex Coplan's user avatar
2 votes
4 answers
930 views

How to scale down for low memory server?

I'm having problem with one of my Rails applications, that runs on 365MB VPS, which seems to be not very much. When there are more than 3-5 users at the same time it runs out of memory and starts to ...
Jakub Arnold's user avatar
  • 1,804
2 votes
3 answers
4k views

MySQL Hardware Specs

Currently we have a database that is 35gb in size and is running on a Virtual Machine with 2GB of RAM. The database is expected to increase at a rate of 3 to 4Gb per day continuously. What are the ...
Jonar's user avatar
  • 171
2 votes
3 answers
1k views

MySQL : table organisation for very large sets with high update frequency

I'm facing a dilemma in the choice of my MySQL schema application. So before I start here is a picture extremely simplified of my database : Schema here : http://i43.tinypic.com/2wp5lxz.png In one ...
Remiz's user avatar
  • 135
2 votes
2 answers
6k views

How to use a distributed PostgreSQL database on a web site?

I want to host a web site, the back end will be a PostgreSQL database server. I would like to have multiple web servers, in different data centers for redundancy. I would then round robin DNS each web ...
Anthony D's user avatar
  • 145
2 votes
1 answer
1k views

Google App Engine vertical scaling

I know Google Cloud (Google App Engine) can autoscale instances horizontally: you have 1 server with certain parameters, when it's overloaded autoscaler launches new server and balances load between ...
pp_1's user avatar
  • 101
2 votes
3 answers
5k views

Best way to scale up an IIS/SQL Server application

We have an application that we developed in ASP and SQL Server. We use Rackspace to host it. Each of our "customers" has their own IIS site and SQL database. Each customer could have a dozen or ...
user avatar
2 votes
4 answers
628 views

What should I do to scale OUT a Sql2008 database?

I'm about to scale UP our sql2008 database. Simple. But I might need to scale OUT our sql databases. For a simple scale out situation (ie distributing the processing load), are there some good ...
Pure.Krome's user avatar
  • 6,628
2 votes
1 answer
127 views

Thought about using my server hardware effectively

I will receive my new server hardware in some days, and of course I am already planning on how to use it the most effective way. My current thoughts are about scaling the web and database server on a ...
Niklas S.'s user avatar
  • 167