Questions tagged [scaling]
The scaling tag has no usage guidance.
142
questions
105
votes
8
answers
221k
views
What limits the maximum number of connections on a Linux server?
What kernel parameter or other settings control the maximum number of TCP sockets that can be open on a Linux server? What are the tradeoffs of allowing more connections?
I noticed while load testing ...
- 2,396
48
votes
5
answers
99k
views
How many requests should my webserver be able to handle?
Not going into specifics on the specs since I know there is no real answer for this. But I've been doing load testing today with the ab command in apache.
And got to the number of 70 requests per ...
- 686
22
votes
2
answers
18k
views
How can I make Microsoft Management Console high-DPI aware in Windows 8.1?
Since Windows 8.1 doesn't allow system-wide "Windows XP style" high DPI support, how can I make the Microsoft Management Console apps (mmc.exe) high-DPI aware? There is no "Troubleshoot compatibility" ...
- 385
20
votes
4
answers
10k
views
Which database server scales better: PostgreSQL or MySQL?
I'm working on a website that could eventually have a very large database. The website will be an art community not until deviantART. I'll be running a lot of queries for analyzing how the site is ...
- 639
19
votes
4
answers
19k
views
How to allow elastic load balancer through port 80 in security groups?
I want to temporarily block port 80 for the outside world, but want the load balancer to pass through the firewall (via security group) so it won't see the instance as unhealthy. How can I do that?
...
- 431
17
votes
3
answers
9k
views
Is there a way to add more backend server to haproxy without restarting haproxy?
We want to be able to add more backend servers on demand. Right now I don't see a way to add more backend servers to the config file without restarting haproxy.
- 2,403
16
votes
2
answers
19k
views
Tips for maximizing Nginx requests/sec?
I'm building an analytics package, and project requirements state that I need to support 1 billion hits per day. Yep, "billion". In other words, no less than 12,000 hits per second sustained, and ...
- 446
13
votes
5
answers
842
views
How do services with large ingest rates install enough hard disks fast enough?
Amazon, Facebook and Google must take in TBs if not PBs per day, so does that mean they have people continuously installing new hard disks and cables in new racks in the same way you might build a ...
- 2,635
11
votes
5
answers
14k
views
AWS: Multi-region setup using single RDS instance
I'm trying to scale our web application (PHP, MySQL, memcache) in a multi-region scheme. Currently we are using a setup with two EC2 instances behind an ELB and an RDS instance, all of them in US-EAST ...
- 278
10
votes
6
answers
14k
views
How to identify heavy write to disk?
I have this problem with server running CakePHP application. The server is insanely slow, I first thought that it's application problem, but then I found constant 5-6MB/s write to disk.
What is the ...
- 1,804
10
votes
1
answer
7k
views
Recommendations for Elastic Search hardware [closed]
Are there any good guides for hardware level to support ElasticSearch? Are recommendations for Lucene or Solr a good place to start? We're looking at rolling out a deployment starting with
27 million ...
- 201
9
votes
8
answers
1k
views
More RAM vs. more servers
I was recently asked "Do you know when to decide between going for more RAM or more servers?" (in the context of scaling data mining applications).
I had no idea, so what are some ways to decide? I ...
user357972
9
votes
3
answers
9k
views
Postgres vs MySQL: Which one is more difficult to scale out?
From your experience, which database serve is more difficult to scale? MySQL or Postgres? I know MySQL has several out-of-the-box scaling/clustering features, but with Postgres there is none that ...
- 485
9
votes
2
answers
2k
views
Large VPN network (~600 servers) with OpenVPN
I'm doing a preliminary study for a contract to build a VPN network between ~600 remote servers running Linux CentOS 6 (+ their 600 private LANs). The network is supposed to be star-based, so that ...
- 425
9
votes
3
answers
2k
views
The best way to share a file uploads directory across horizontally scaling web servers
I'm currently trying to spec up a horizontally scalable cluster for a drupal based web app, that looks something like the colorful diagram below:
The load balancer implements sticky sessions, so a ...
- 709
8
votes
10
answers
26k
views
Where to purchase a server capable of 512GB of RAM? [closed]
Where can I purchase a single server capable of 512 GB, yes GB, of RAM?
The most I've seen is 256 GB (32 DIMM x 8 GB per stick)
Timk
7
votes
4
answers
7k
views
Load balanced Drupal servers: a good way to share data among the servers?
I'm trying to get an overview of how i would go about creating a load balanced web server setup. Setting up the actual load balancer and adding two or more web servers seems fair enough. However, I ...
- 1,630
7
votes
1
answer
2k
views
How do I scale my EC2 instance?
I'm very new to server configurations and Linux in general, so please bear with me:
I have a small EC2 instance running a LAMP server on Ubuntu 10.04 LTS.
I'm completely baffled as to how I would ...
- 413
7
votes
1
answer
10k
views
How to horizontally scale SSL termination behind HAProxy load balancing?
I've been looking around and no one seems to be trying to scale SSL termination the way I am and I'm curious as to why my approach seems so uncommon.
Here's what I want to do followed by why:
10.0....
- 181
7
votes
3
answers
702
views
how to go about scaling a web-application?
I am primarily a web-application developer, and I do not know much about scaling/scalability techniques. My application is written in Python, using Django; a fairly standard setup.
I currently use ...
- 179
7
votes
2
answers
4k
views
HAProxy not balancing server load fairly
HAProxy doesn't seem to keep the connections to the servers balanced.
Keep this in mind:
using HAProxy v1.3.26
5 equally balanced server specs
algorithm is round robin, but no weights are applied ...
- 71
6
votes
2
answers
1k
views
AWS - how to serve faster to oversea users
We have our infras relying on AWS in the us-east-1 region. (EC2, CloudFront, RDS, ElastiCache)
We are now having more and more users from the APAC. Users start to complain about network speed to our ...
- 283
5
votes
6
answers
5k
views
How can I have 2 database servers for my website?
I have a site that has heavy traffic and my database server (mysql) is getting heavy traffic at peek hours. Instead of upgrading to a better server, I am thinking of having 2 database servers working ...
- 51
5
votes
5
answers
4k
views
When is the right time to add a separate database server, add more web servers?
Usually web projects start small, everything is on the one server. But if site becomes popular one server won't be enough.
So, since I don't have any experience in this, I'd like to get an idea of:
1. ...
- 301
5
votes
1
answer
590
views
Sharding MongoDB internationally with data stored based on location
We're looking to set up data centers globally and switch to a NoSQL database like MongoDB to support having our data distributed over these data centers (and as a pleasant side effect, introduce more ...
- 153
5
votes
2
answers
1k
views
Memcached scaling strategy
Currently I am running a production environment with 4 dedicated memcached servers, each of them having 48Gb of RAM (42 dedicated to memcache). Right now they are doing fine, but traffic and content ...
- 402
4
votes
3
answers
6k
views
Suggestions on large scale web applications architecture?
It's a big question:) We are running a website with LAMP that is not big, 5 web servers with LVS load balancing, 3 MySQL servers with replications and separation of reading and writing, and we use ...
- 961
4
votes
4
answers
19k
views
Maximizing TCP connections on HAProxy load balancer
I am currently using HAProxy in order to load balance tcp connections from clients to my Erlang app server. The connection is persistent, which means I'm limited to roughly 64K clients on an optimized ...
- 1,991
4
votes
4
answers
777
views
Multiple web servers sharing a db server
I am contemplating migrating to mosso to have several cheap app servers.
The goal is to have a dedicated database server, however, does it make sense to have one database server that hosts the ...
- 141
4
votes
5
answers
2k
views
HAProxy and "sharding"
I am cross posting this from stack overflow since it was suggested I might get a better answer here ...
I was wondering if anyone has used HAProxy for sharding. Specifically, I would like to be able ...
Manish V
4
votes
1
answer
2k
views
Creating a stable, auto-scaling app server cluster
I have two servers, each running anywhere from 50-100 different legacy webapps, written in languages ranging from PHP to Python to Ruby-on-rails to NodeJS.
We're wanting to kill off these machines ...
- 177
4
votes
1
answer
3k
views
Can I use MongoDB with Amazon EC2 small instance?
I want to move my MongoDB from a virtual server to amazon web services. I know MongoDB provides a preconfigured AMI, with three provisioned EBS volumes. But this is only available for m1.large ...
- 522
4
votes
3
answers
217
views
First steps in scaling servers up for a Ruby on Rails application
I'm running a RoR web application on a dual core AMD Athlon 4450e server w/ 8GB RAM and Ubuntu.
It's doing alright, getting spikes up to 320 dynamic requests per minute, it's also handling all the ...
- 195
3
votes
11
answers
5k
views
Active Directory for web authentication: scale to 1M users?
I'm interested in how well Active Directory would fare as the authentication backend for a website, scaled for ~1 million users. Do you have experience with AD in web environments of this scale, and ...
- 748
3
votes
4
answers
2k
views
How to choose size for a cloud server (rackspace)
We're going to test the rackspace cloud next week to see how it's working with our web app. It's a LAMP environment with a lot of MySQL databases.
How do I choose the "right" server size? On ...
- 133
3
votes
1
answer
462
views
Firewall as a guest VM?
Out of curiosity, would it be pointless/wasteful/silly to put a firewall as a VM guest (regardless of VM host - ESX, Xen, Hyper-V, etc. etc.) and redirect all traffic from other VM guests through the ...
- 3,875
3
votes
2
answers
270
views
Virtualization as a way for scaling
I'm asking this question to find out what are you best practices to prepare your system for scaling, I know that basically you don't have to over optimized before really having to, and that's the best ...
- 1,158
3
votes
2
answers
3k
views
horizontal scaling and load-balancing for PostgreSQL with ProxySQL
I have a PostgreSQL DB running in Google Cloud SQL that I need to scale horizontally. Cloud SQL makes it easy to add read-only replicas, but does nothing to help with load-balancing traffic to the ...
- 297
3
votes
2
answers
304
views
RT database scaling
Recently I heard someone suggest that RT request tracker may have scalability issues due to its non-normalized database (someone at a Perl meeting I went to referred to it in a positive light as hyper-...
- 131
2
votes
7
answers
6k
views
How to deploy web application for 2 million users?
I've been tasked with figuring out how to do a large scale deployment for a high availability, high traffic, dynamic, flash based web application. There is a good possibility that this application can ...
- 366
2
votes
2
answers
3k
views
Why is my AWS instances suddenly becoming irresponsive reporting high "stolen" CPU
The setup
I have a bunch of t2.small EC2 instances running hosting the image processing library called thumbor for simple on-the-fly image resizing. Originals are loaded from S3. In front of the ...
- 358
2
votes
3
answers
1k
views
How do multiple servers work in terms of databases? [closed]
I don't know an awful lot about servers, but I do know that there are many ways you can improve the performance of running a web application, in terms of hardware.
For example, starting with a single ...
- 595
2
votes
4
answers
930
views
How to scale down for low memory server?
I'm having problem with one of my Rails applications, that runs on 365MB VPS, which seems to be not very much. When there are more than 3-5 users at the same time it runs out of memory and starts to ...
- 1,804
2
votes
3
answers
4k
views
MySQL Hardware Specs
Currently we have a database that is 35gb in size and is running on a Virtual Machine with 2GB of RAM. The database is expected to increase at a rate of 3 to 4Gb per day continuously.
What are the ...
- 171
2
votes
3
answers
1k
views
MySQL : table organisation for very large sets with high update frequency
I'm facing a dilemma in the choice of my MySQL schema application. So before I start here is a picture extremely simplified of my database :
Schema here : http://i43.tinypic.com/2wp5lxz.png
In one ...
- 135
2
votes
2
answers
6k
views
How to use a distributed PostgreSQL database on a web site?
I want to host a web site, the back end will be a PostgreSQL database server. I would like to have multiple web servers, in different data centers for redundancy. I would then round robin DNS each web ...
- 145
2
votes
1
answer
1k
views
Google App Engine vertical scaling
I know Google Cloud (Google App Engine) can autoscale instances horizontally: you have 1 server with certain parameters, when it's overloaded autoscaler launches new server and balances load between ...
google-cloud-platform
2
votes
3
answers
5k
views
Best way to scale up an IIS/SQL Server application
We have an application that we developed in ASP and SQL Server. We use Rackspace to host it. Each of our "customers" has their own IIS site and SQL database. Each customer could have a dozen or ...
user35697
2
votes
4
answers
628
views
What should I do to scale OUT a Sql2008 database?
I'm about to scale UP our sql2008 database. Simple. But I might need to scale OUT our sql databases.
For a simple scale out situation (ie distributing the processing load), are there some good ...
- 6,628
2
votes
1
answer
127
views
Thought about using my server hardware effectively
I will receive my new server hardware in some days, and of course I am already planning on how to use it the most effective way.
My current thoughts are about scaling the web and database server on a ...
- 167