Iâve done this myself, figuring weâd kick off the project and show how we
could âoptimizeâ to local disks later. Let me say this unequivocally: You
absolutely should not use a SAN or NAS with Hadoop.
Did not account for increased latency after moving to EC2. In the datacenter they had submillisecond access between machines so it was possible to make a 1000 calls to memache for one page load. Not so on EC2. Memcache access times increased 10x to a millisecond which made their old approach unusable. Fix was to batch calls to memcache so a large number of gets are in one request.
I was confused about the memcached problem after moving to the cloud.Â I understand why network latency may have gone from submillisecond to milliseconds, but how could you improve latency by batching requests? Shouldn’t that improve efficiency, not latency, at the possible expense of latency (since some requests will wait on the client as they get batched)?Â
Jeremy cleared it up by saying:
The latency didn’t get better, but what happened is that instead of having to make a lot of calls to memcache it was just one (well, just a few), so while that one took longer, the total time was much less.
ButÂ Dave RosenthalÂ created a great graphic showing how batching can in fact decrease total system latency:
Editor’s note: This is a slightly modified version of a previous post.
Job hunting season is upon us. Openings are already being posted here, here, and here. So you should have your CV, research statement, and web page ready. I highly recommend having a web page. It doesnât have to be fancy. Here, here, and here are some good ones ranging from simple to a bit over the top. Minimum requirements are a list of publications and a link to a CV. If you have written software, link to that as well.
The earlier you submit the better. Donât wait for your letters. Keep in mind two things: 1) departments have a limit of how many people they can invite and 2) admissions committee members get tired after reading 200+ CVs.
If you are seeking an academic job your CV should focus on the following: PhD granting institution, advisor (including postdoc advisor if you have one), and papers. Be careful not to drown out these most important features with superflous entries. For papers, include three sections: 1-published, 2-under review, and 3-under preparation. For 2, include the journal names and if possible have tech reports available on your web page. For 3, be ready to give updates during the interview. If you have papers for which you are co-first author be sure to highlight that fact somehow.
So what are the different types of jobs? Before listing the options I should explain the concept of hard versus soft money. Revenue in academia comes from tuition (in public schools the state kicks in some extra $), external funding (e.g. NIH grants), services (e.g. patient care), and philanthropy (endowment). The money that comes from tuition, services, and philanthropy is referred to as hard money. Within an institution, roughly the same amount is available every year and the way its split among departments rarely changes. When it does, itâs because your chair has either lost or won a long hard-fought zero-sum battle. Research money comes from NIH, NSF, DoD, etc.. and one has to write grants to raise funding (which pay part or all of your salary). These days about 10% of grant applications are funded, so it is certainly not guaranteed. Although at the institution level the law of large numbers kicks in, at the individual level it certainly doesnât. Note that the break down of revenue varies widely from institution to institution. Liberal arts colleges are almost 100% hard money while research institutes are almost 100% soft money.
So to simplify, your salary will come from teaching (tuition) and research (grants). The percentages will vary depending on the department. Here are five types of jobs:
1) Soft money university positions: examples are Hopkins and Harvard Biostat. A typical breakdown is 75% soft/25% hard. To earn the hard money you will have to teach, but not that much. In my dept we teach 48 classroom hours a year (equivalent to one one-semester class). To earn the soft money you have to write, and eventually get, grants. As a statistician you donât necessarily have to write your own grants, you can partner up with other scientists that need help with their data. And there are many! Salaries are typically higher in these positions. Stress levels are also higher given the uncertainty of funding. I personally like this as it keeps me motivated, focused, and forces me to work on problems important enough to receive NIH funding.
1a) Some schools of medicine have Biostatistics units that are 100% soft money. One does not have to teach, but, unless you have a joint appointment, you wonât have access to grad students. Still these are tenure track jobs. Although at 100% soft what does tenure mean? I should mention at MD Anderson, one only needs to raise 50% of ones salary and the other 50% is earned via service (statistical consulting to the institution). I imagine there are other places like this, as well as institutions that use endowments to provide some hard money.
2) Hard money positions: examples are Berkeley and Stanford Stat. A typical break down is 75% hard/25% soft. You get paid a 9 month salary. If you want to get paid in the summer and pay students, you need a grant. Here you typically teach two classes a semester but many places let you âbuy outâ of teaching if you can get grants to pay your salary. Some tension exists when chairs decide who teaches the big undergrand courses (lots of grunt work) and who teaches the small seminar classes where you talk about your own work.
2a) Hard money postions: Liberal arts colleges will cover as much as 100% of your salary from tuition. As a result, you are expected to teach much more. Most liberal arts colleges weigh teaching as much (or more) than research during promotion although there is a trend towards weighing research more.
3) Research associate positions: examples are jobs in schools of medicine in departments other than Stat/Biostat. These positions are typically 100% soft and are created because someone at the institution has a grant to pay for you. These are usually not tenure track positons and you rarely have to teach. You also have less independence since you have to work on the grant that funds you.
4) Industry: typically 100% hard. There are plenty of for-profit companies where one can have fruitful research careers. AT & T, Google, IBM, Microsoft, and Genentech are all examples of companies with great research groups. Note that S, the language that R is based on, was born in Bell Labs. And one of the co-creators of R now does his research at Genentech. Salaries are typically higher in industry and cafeteria food can be quite awesome. The drawbacks are no access to students and lack of independence (although not always).
5) Government jobs: The FDA and NIH are examples of agencies that have research positions. The NCIâs Biometric Research Branch is an example. I would classify these as 100% hard. But it is different than other hard money places in that you have to justify your budget every so often. Service, collaborative, and independent research is expected. A drawback is that you donât have access to students although you can get joint appointments. Hopkins Biostat has a couple of NCI researchers with joint appointments.
Ok, that is it for now. Later this month we will blog about job interviews.
Over the years, I have mentored quite a few System Administrators. Levelling up means learning about your tools and what theyâre capable of (and not memorizing command line flags). For this yearâs article on SysAdvent, I wanted to share a lot about one my favorite tools: yum. When I say yum, I mean a little more than just the yum cli itself, but the ecosystem of tooling around it. I spend a lot of time doing things like package building, package repository management, and all in all hacking around with rpms and yum.
Yum is a tool that youâve probably used if you been a system administrator for any period of time. Itâs also one of those tools that is very easy to use and have it get out of your way. yum does network-based dependency resolution, meaning that if you want to install a package, it will download and install all dependencies of that package as well. These are the basics people often know. Under the hood it uses rpm. In normal operation, you use yum for searching, installation and uninstallation of packages. Thatâs actually pretty awesome, but mainly the trivial use-case for yum.
Beyond that, however, there is much more to the way yum works and interacts with repository metadata. Sometimes being able to query that data can solve heinous problems easily, rather than coming up with odd workarounds. That information can also help you make good decisions about package management.
So, rather than bore you with more background about the yum you already know, letâs dive into some examples. Some of these examples are certainly tailored for beginners, and some are more advanced. I hope the quantity of examples will make this post a good reference.
Yum client tips
Letâs get started on the client side of yum — meaning just the CLI utility, yum. Beyond the normal update, erase, install there are a few nice enhancements that you may not be aware of.
If youâve ever wanted a summary of what updates and types are available for your system, try the updateinfo subcommand.
In this case, the security update looks like something I should probably install. The other fixes I just donât have time to vet right now. If you only want to install the security updates, just run âyum update --securityâ. There are also specific options for Common Vulnerability Exposures CVEs and Red Hat Security Advisories (RHSA) if you follow those types of announcements and want to be very specific about updates you take on.
Yum also provides a nice way to list out the repositories it knows about, and how many packages are in each repository.
stahnma@hu ~> yum repolist
repo id repo name status
base CentOS-6 - Base 6,381
cr CentOS-6 - CR 1,215
epel Extra Packages for Enterprise Linux 6 - x86_64 10,081
extras CentOS-6 - Extras 13
puppetlabs-pepackages Puppet Labs PE Packages 6 - x86_64 58
updates CentOS-6 - Updates 1,555
If youâre on a multi-user system, yum history is a bit fun to see whoâs been installing packages.
root@f3 ~ # yum history
ID | Login user | Date and time | Action(s) | Altered
64 | root <root> | 2013-11-27 03:45 | Update | 1
63 | root <root> | 2013-11-25 03:28 | E, I, U | 11
62 | root <root> | 2013-11-24 03:22 | Update | 1
61 | root <root> | 2013-11-21 03:23 | Update | 4
60 | root <root> | 2013-11-20 03:18 | Update | 4
59 | root <root> | 2013-11-19 03:35 | Update | 1
58 | root <root> | 2013-11-18 03:33 | E, I, U | 6
57 | Michael ... <stahnma> | 2013-11-17 07:49 | Install | 2
Seeing root performing a majority of the action may be expected. In the case where you prefixed the yum command with sudo as a normal user, it tracks that as well. The E, I, U is Erase, Installed and Updated which were the actions performed.
Similarly, when you want to see where a file came from on an already installed system, you can use yum. This can be important when you have a package that depends on a specific file rather than a package name. Here we get the package it came from, the version and the repository it was installed from.
Rather than use resolvedep, if you know the filename youâre looking for, you can just install a package based off of a file in its path. As an example, in a past life, I needed to have uuencode available on several systems. For the life of me, I could never remember what package uuencode was supplied in because it wasnât uuencode.rpm.
What I used to do was find a system that already had uuencode on it, and run an rpm query.
That obviously didnât work if I couldnât find a system that already had uuencode on it. Luckily with yum, I can just say, “hey, I want that command installed.”
stahnma@hu ~> sudo yum install /usr/bin/uuencode
Loaded plugins: fastestmirror, security
Setting up Install Process
--> Running transaction check
---> Package sharutils.x86_64 0:4.7-6.1.el6 will be installed
--> Finished Dependency Resolution
Package Arch Version Repository Size
sharutils x86_64 4.7-6.1.el6 base 187 k
Install 1 Package(s)
Total download size: 187 k
Installed size: 617 k
Is this ok [y/N]:
Ah-ha, the package was sharutils. What if you just wanted to find out what package gave you /usr/bin/uuencode without starting a yum package transaction? It turns out that all of this information is readily available in the metadata of the package repositories yum connects to. To query that metadata, you use the appropriately named command, repoquery, which is included in the yum-utils package.
Here, you get the results of what includes that command, without starting a package installation transaction.
What if you want to see what sharutils is, and what files it comes with? repoquery can help you here very easily. Itâs also worth noting that repoquery can be run with a non-privileged user account.
stahnma@hu ~> repoquery -il sharutils`
Name : sharutils
Version : 4.7
Release : 6.1.el6
Size : 631381
Packager : CentOS BuildSystem <http://bugs.centos.org>
Group : Applications/Archiving
URL : http://www.gnu.org/software/sharutils/
Repository : base
Summary : The GNU shar utilities for packaging and unpackaging shell archives
Source : sharutils-4.7-6.1.el6.src.rpm
The sharutils package contains the GNU shar utilities, a set of tools
for encoding and decoding packages of files (in binary or text format)
in a special plain text format called shell archives (shar). This
format can be sent through e-mail (which can be problematic for regular
binary files). The shar utility supports a wide range of capabilities
(compressing, uuencoding, splitting long files for multi-part
mailings, providing checksums), which make it very flexible at
creating shar files. After the files have been sent, the unshar tool
scans mail messages looking for shar files. Unshar automatically
strips off mail headers and introductory text and then unpacks the
The -i tells repoquery to provide the package information it can find. The -lis just to list all the files the package contains. In general, the arguments to repoquery are identical to rpm, with the exception that the -q from rpm is implied.
repoquery is like a swiss army knife for yum-based repositories. Here are a few quick examples using repoquery to learn about the packages in our repositories.
In addition to the above examples, repoquery has several more available options and questions it can help answer. However, there are some other tools that do more specific things than repoquery.
Sometimes, you want to look inside of a package. Normally you can do this from repoquery, but for some functions you just want it on disk. Check out yumdownloader for that. I often use yumdownloaderwith the --source option to download the source rpm because I plan to modify it and rebuild it.
If youâve ever had to manage lots of EL systems, you find yourself needing to add on third-party repositories. I have often wanted to know where a package came from. Luckily, a few years ago, find-repos-of-install was written and added to yum-utils.
stahnma@hu ~> find-repos-of-install
xz-5.1.2-4alpha.fc19.x86_64 from repo koji-override-0
xz-devel-5.1.2-4alpha.fc19.x86_64 from repo fedora
xz-libs-5.1.2-4alpha.fc19.x86_64 from repo koji-override-0
ykpers-1.13.0-1.fc19.x86_64 from repo updates
In this case, I have fedora-updates-testing enabled and some custom koji (the Fedora build system) repositories.
Other times, youâll want to play the role of an archaeologist and figure out how or why a package is installed on a system. Luckily, with the yumdb, you can figure that out pretty easily. yumdb does need to be run as root.
root@f3 ~ # yumdb get reason ack perl
reason = user
reason = dep
This shows that ack was installed because it was explicitly called out by a user. perl, however, was pulled in as dependency of something else that a user specified. If youâd like to see what else is in the yumdb for a package, try yumdb info <packagename>.
Have you ever noticed how when you run a GUI on linux, after updates it alerts you if you need to restart something, or reboot? When running headless, you donât have that nice reminder, but you do still have the logic behind it. After you perform updates, you might need to restart some daemons (or even the whole system). Enter the command needs-restarting.
The command provides a list of PIDs and programs path names that started before components they rely upon were updated.
Weâve dug into a few of the ways that some programs inside the yum-utils package can give you more insight into items on your local system. What if we start to look to look at the repositories as a whole?
Now we can start to walk through analysis of the metadata and repositories. This can lead to knowing a whole lot more about how your systems are put together. Dependencies for yum are tracked in a directed (and normally acyclic) graph. If youâve ever wanted to see the graph from a yum repository, you can. You can even see one of all repositories combined (but it will be HUGE and everything will point to glibc). Here you can see the graph of the puppetlabs-products repository found on http://yum.puppetlabs.com. In this use-case, I already have the *puppetlabs-products * repository setup on my system. To see what repositories are installed, try yum repolist.
This took about 20 seconds to download all of the Fedora 19 x86_64 rpms from in the repository. reposync did not create the rpm metadata for me. I would have to do that manually or in a cron job. Also, if I wanted more than just fedora-19-x86_64, I would have to make special configuration file for this mirroring.
For larger tasks and mirrors, making a configuration file is ideal. The configuration file needs to have a [main] section and define reposdir or else youâll pull in the entire systemâs yum configuration in addition to the repositories defined in your configuration file.
This configuration file will mirror EL6 and Fedora 19 x86_64. It also only pulls the newest items and does include source rpms.
If Iâve been operating a mirror or package repository for a while, there is a good chance that I have lots of older rpms around that will never get pulled in by yum anymore. What Iâd like to do is remove old rpms if there are newer ones that replace them. yum-utils includes repomanage to do that. repomanage is able to list out the packages which wonât be utilized by yum because they have been replaced by newer versions of the package.
Now my mirror is eating a little less disk space, but it still takes up a lot of space and network bandwidth to get, especially on large repositories like EPEL or Fedora. Also, there lots of packages in those repositories that I will likely never use. Luckily there is repotrack available.
repotrack takes a package as an argument and then will mirror that package and all its dependencies. Even the smallest package normally has several layers of indirect dependencies. For example, perl has 38 in a Fedora 19 system, git has 170.
If I want to mirror a few packages (and all deps to be sure it installs cleanly from my repository) I would do something like this.
stahnma@f3 ~> repotrack perl git
I now have a perl, git and all of their dependencies downloaded into a single directory that I can use as yum repository.
Now, over time, I will likely find yourself adding packages, replacing packages, and trimming out older packages. With this type of customization (especially if you run your own yum repositories), I will want to make sure the dependencies can still be resolved and I havenât put my users in an impossible state. To solve this problem, there is repoclosure.
repoclosure, by default, will source yum.conf and see if I have any unmet dependencies in the repositories enabled on the system. I recommend having a custom configuration for this command, much like the one used in the reposync example. The main difference here is that I also want to include the base system repositories for a complete closure.
For example, take the puppet rpm, it depends on a few things in the puppetlabs yum repositories, but those, in turn, depend on items in the base repositories (such as ruby). Without having the base repositories specified in my configuration file, I would never get a clean closure of the puppetlabs-products-el-6-x86_64 repository.
# This configuration file is at https://gist.github.com/stahnma/7739439
stanma@f3 ~> repoclosure -c el6 -t
Reading in repository metadata - please wait....
Repos looked at: 4
Num Packages in Repos: 8295
Weâve seen a few of the helpers that yum-utils provides to make managing repositories a little easier. We can mirror entire repositories, track just a package and its dependencies, prune older content, and ensure all dependencies are resolvable given a known set of repositories. If you do all of these things, your custom yum repository will likely be in pretty good shape.
DNF - the future
If you havenât already had enough fun learning about yum and its tool set, thereâs a new tool on the horizon with a longer term goal of displacing yum called dnf. Itâs a next generation yum tool that is nearly 100% abi/api compatible with yum. It is however, faster. To accomplish this, speed increase it uses libsolv (from zypper fame if youâre a SuSE fan) under the hood. dnf is available in Fedora 18 and higher, but not installed by default. You may wish to play with it if you want to see the future of where yum and repodata are headed.
I hope you enjoyed this quick walk through some of the fun and power you can have when working with yum and its associated repository information. Once you can easily query your package manager and repositories, you should be able to head down a path of better times, including automation, and rapid prototyping. Knowing this type of information is also critical when you start making your own packages and repositories.
Lastly, Iâd like to thank Seth Vidal for his contributions to making us all better at managing our systems because of the tools he wrote. You are missed.
Oregon has spent more than $40 million to build its own
online health care exchange. It gave that money to a
Silicon Valley titan, Oracle, but the result has been a
disaster of missed deadlines, a nonworking website and a
state forced to process thousands of insurance
applications on paper.
Some Oregon officials were sounding alarms about the tech
companyâs work on the stateâs online health care exchange
as early as last spring. Oracle was behind schedule and,
worse, didnât seem able to offer an estimate of what it
would take to get the stateâs online exchange up and
The biggest reason Cover Oregonâs website lags behind is
because Oracle didnât meet its deadline and should have
begun testing last May, rather than delaying until this
summer when it was too late to resolve the problems it
encountered, King said.
Oracle has been paid handsomely by Cover Oregon for its
consulting and software development. Itâs received $43.2
million this year â accounting for $11.1 for hardware,
$9.5 million for software and $22.6 million for
So even if you use Oracle for everythingâhardware, software, and consulting payed with a paltry $43.2mil in 2013, you can still fail? What a surprise!
â Whoâll take the blame if HealthCare.gov and Cover Oregon would just switch their contractors?
â Could we have these played on repeat for those blaming MarkLogic for HealthCare.govâs failure? Also for those that accepted this excuse?
Site reliability engineers in the most general sense are charged with a clear mission: efficiently keep the sites reliable. Reliability can be broken down into two main facets, availability and performance. This is about where it stops being straightforward and everything becomes nuanced. This is because you have to startdefining what availability and performance means for your systems (which is generally driven by the mission of your organization and how your systems fit into that). Even more complexity comes into play when you consider all the activities an SRE team engages in to achieve these things. For example configuration management, capacity planning, restores, fault tolerance, and security to name some of them.
How you define availability and performance in your organization is a topic worthy of its own set of posts and the details of all the activities an SRE team participates could fill a library. An SRE team needs to start somewhere and have a strategy to tackle all of this. There is no one answer, but achieving a high level observability needs to be a key strategic component for any SRE team.
Observability is the Foundation
Observability is the degree and facility in which your team can gain insight into the behavior of your systems. It is worth noting that the scope of your systems is likely quite broad, it includes the obvious things like your applications and hosts, but also includes things like processes, workflows, and team dynamics. Having insight in your systems means:
Questions operators have about their systems can be quantifiably answered with minimal effort
Operators have rich mental models of how their systems function
When you have to decided something you can either guess, or use âthe scienceâ. Without a set of systems for observability in place you will end up guessing (not the educated kind) or be terribly inefficient. A good understanding of how systems work is what allows operators to be effective and avoid disastrous mistakes, and observability can drive that.
Decision Making and Incident Preparedness
Observability is key to the strategy for an SRE team because it informs and impacts nearly every other activity that team engages in. Iâve written about the OODA loop before which stands for Observe, Orient, Decide, Act (You can think of Orient as âAnalyzeâ). It is a military strategy that suggests you can be successful when you can rapidly and successfully iterate through this loop quickly. It is also a tool that is useful for thinking about site reliability operations as well.
OODA is carried out at both the macro and micro levels (planning and incidents) by SRE teams. As an example, we can imagine what making system design decisions as a team is like without good observability (and since we have likely all been there, you can probably just remember). The observation phase will be based on peopleâs memory and is frequently skipped. Orienting or analyzing that information as a group will have conflicts because people donât agree on what the facts are. This can result in arguments about the personâs recollection of the facts instead of the issue at hand. Decisions end up being prolonged and half hearted because of the uncertainty of their basis. Lastly, action will be hindered because a strong consensus hasnât been reached because people donât trust the baseless decision. Even worse, people question if this is even the system they should be working on at all.
Many have also probably been through outages when observability is lacking. Lots of time is lost trying to figure out what is even going on. Orienting is difficult because operators lack the internal model of the system that observability provides over time. As a result of these things decisions and actions are chaotic. Or more simply put, itâs amateur hour.
In contrast, the picture is entirely different with a solid foundation in observability because everything becomes data informed. This is different from âdata drivenâ because you can trust peopleâs intuition. Due to good observability they have developed keen instincts about systems over time. When it comes to system design decisions you are in a much better position because chances are you are designing the right thing in the first place. Team members will bring their observations to the discussion. If there are questions about the facts, instead of arguing then you can just look them up. Decisions will be made with more confidence and faster because they are based on evidence. Lastly, action will have more consensus behind it, even if people didnât agree they at least know the choice was based on something.
You never know what the next incident will be, but if you have good observability then your operators will have a deeper understanding of the system and will be far more prepared for the unknown.
Observability positions a team to do more capacity planning by enabling them to see constrained resources and forecast growth. This can help reduce the vicious cycle of fire fighting that many SRE teams are locked into.
Since observability leads to insight, team members are learning more about their systems which generally is a common source of fulfillment for engineering types.
Convinced? 5 Steps to Achieving Good Observability:
In order to achieve good observability an SRE team (often in conduction with the rest of the organization) needs to do the following steps.
Instrument your systems by publishing metrics and events
Gather those metrics and events in a queryable data store(s)
Make that data readily accessible
Highlight metrics that are or are trending towards abnormal or out of bounds behavior
Establish the resources to drill down into abnormal or out of bounds behavior
Each of these steps largely depends on the previous step to be successful.
1. Instrument your Systems
Brainstorm what key and useful metrics exist for your system. Make those metrics easily accessible (i.e. standard APIs like json via REST or by providing a destination to push to) and document what they are and what the implications of those metrics are. This largely falls on the developers of systems, and DevOps culture can go a long way encourage application developers to empower the operations side of things by doing this. At the highest level you can break metrics and events into two categories:
Objective Oriented: These metrics reflect the mission of your organization. For example they include client facing measurements like response time, availability, error codes, items sold, number of users, number of active user and content created
Diagnostic Oriented: These measure aspects of the system that allow you to achieve your objects. These include system measures such as OS, network, hardware, middleware, cluster, and application metrics. These also include response time and availability metrics but they measure components and parts of the pipeline that contribute to your objectives.
Good Metrics also tend to have these properties:
High Resolution: âHighâ is qualitative, but a higher frequency of data collection means you will have more insight into the shape of your data (i.e. is it bursty)
Lossless: This means that there isnât missing information from your metric. This can often be achieved by publishing counters instead of rates and letting the display side of things calculate a rate from that information. Also not pre-aggregating things into averages can be useful (or if you are going to do that also aggregate the data into multiple percentiles)
Specific: More specific metrics can often be more useful to understanding a system and drilling down into a problem. For example, with something like CPU utilization it is better to report something like %user, %system CPU time breakdowns and let something later in the pipeline aggregate them.
It is also worth making a point to instrument your own internal âmetaâ systems such as bug tracking and documentation.
2. Gather those metrics in a queryable data store(s)
This is a key intermediate step to making this data accessible. Data generally needs to be stored over time in order to give it context (although the time of each datapoint isnât always important for things like histograms when it is processed later). Having this step enables things like:
Enabling capacity planning
Allowing operators to explore the data and learn
Allowing people to invent cool stuff you didnât anticipate
As a rule of thumb, less data stores are better because it makes easier to work with the data (although specialized databases for things like time series might be worth the tradeoff because of features and scalability). For time series data in particular, a couple of useful qualities are:
Scalability: This enables one to collect a lot of metrics, at high resolution, and high retention
Aggregation: This encourages a shift from host/process oriented views to cluster and service oriented views
3. Make that data Readily Accessible
If there is a lot of friction to view the data then people wonât have time or energy to do it. This is why it is important to have good dashboards and APIs to allow easy access for your operators. Good dashboards tend to have the following attributes:
A fast responsive UI to allow for operators to drill down and explore easily
Enables operators to create their own dashboards and graphs
4. Highlight Metrics that are or are trending towards abnormal / out of bounds behavior
Ideally a team ends up collecting a lot of data. This means humans canât process it all. That means your systems need ask for operators attention. Essentially this is alerting. However it is important to understand that alerting doesnât always mean âemailingâ. It can also mean things like publishing something to a dashboard or logging it.
Alert noise / desensitization is a plague in our field, my belief is that future systems will allow for more carefully crafted and adjustable rules to reduce the noise. Keeping this under control is also largely about discipline and remembering that every alert requires action.
5. Establish the resources to drill down into abnormal or out of bounds behavior
The above steps are a gateway to observability. This is because the nature of collecting metrics is resourced constrained. You can only collect so much information without noticeably impacting what you are trying to observe. Eventually you are going to need to drill down into problems or explore further why metrics are behaving in a certain way. There are three common activities for this:
Log analysis: Digging into your system logs for information. System logs can also be a powerful source of metrics (especially things like web logs) if you parse them and feed the results into your monitoring systems
Profiling: This the activity of sampling programs to figure out what they are doing â generally at a much higher resolution than collecting metrics (computer time (sub 1ms) instead of human time)
Tracing: Collecting every single thing a system is doing (i.e. strace or DTrace)
Although my path to observability puts an emphasis on collecting metrics and events, this step is also crucial to observability.
Use the science, Luke
If observability is one of the key components of the strategy for your team, then it sets the tone and foundation for everything else. It can create a culture of constant learning as it provides a medium for learning about your systems and proves a source of information for productive analytical arguments. Whatever your strategy is, you need to consider what role observability plays in your team. And remember: Use The Science.
(Like the new look? Iâve been working with a great designer and will be refreshing the main site too.)
The goal is an intuition-first look at a notoriously gnarly subject. This isnât a replacement for a stodgy textbook â itâs the friendly introduction I wish Iâd had. A few hours of reading that would have saved me years of frustration.
The course text is free online, with a complete edition available, which includes:
The price of the course will increase when the final version is released, so hop onto the beta to snag the lower price.
Building A Course: Lessons Learned
A few insights jumped out while making the course. This may be helpful if youâre considering teaching a course one day (I hope you are).
I struggled with what to make free vs. paid. I love sharing insights with peopleâ¦ and I also love knowing I can do so until Iâm an old man, complaining that newfangled brain-chip implants arenât âreal learningâ.
Incentives always exist. I want to make education projects sustainable, designed to satisfy readers, not a 3rd party.
Similar to the fantastic Rails Tutorial Book, the course text is free, with extra resources available. Having the core material free with paid variations & guidance helps align my need to create, share, and be sustainable.
Being Focused Matters
Historically, Iâm lucky to write an article a month. But this summer, I wrote 16 lessons in 6 weeks. What was the difference?
Well, pressure from friends, for one: Iâd promised to do a calculus course this summer. But mostly, it was the focus of having a single topic, brainstorming on numerous analogies/examples, and carving a rough path through on a schedule (2-3 articles/week).
I hope this doesnât sound disciplined, because Iâm not. A combination of fear (I told people Iâd do this) and frustration (Argh, I remember being a student and not having things click) pushed me. When I finished, I took a break from writing and vegged out for a few weeks. But I think it was a worthwhile trade â in my mind, a yearâs worth of material was ready.
Thereâs many options for making a course. Modules. Quizzes. Interactive displays. Tribal dance routines. Hundreds of tools to convey your message.
Andâ¦ what is that message, anyway? Are we transmitting facts, or building insight?
Until the fundamentals are working, the fancy dance routine seems useless. Iâd rather read genuine insights from a pizza box than have an interactive hologram that that recites a boring lecture.
When lessons are lightweight and easy to update, youâre excited about feedback (Oh yeah! A chance to make it better!).
The more static the medium, the more you fear feedback (Oh no, I have to redo it?). A fixed medium has its place, ideally after a solid foundation has been mapped out.
Iâll be polishing the course in the coming weeks, feedback is welcome!
Note: Taking a page from a bygone era, Henrik Sahlstrom has taken the costumed characters of Gotham City squarely into the realm of noir. Some of the designs go a little bit to far afield of the charactersâ inherent identifiers, all-and-all itâs a wonderful concept that Iâd love to see expanded on. â Chris A.
I like to think of Stack Overflow as runningwith scale but notat scale. By that I meant we run very efficiently, but I still donât think of us as âbigâ, not yet. Letâs throw out some numbers so you can get an idea of what scale we are at currently. Here are some quick numbers from a 24 hour window few days ago â November 12th, 2013 to be exact. These numbers are from a typical weekday and only include our active data center â what we host. Things like hits/bandwidth to our CDN are not included, they donât hit our network.
148,084,883 HTTP requests to our load balancer
36,095,312 of those were page loads
833,992,982,627 bytes (776 GB) of HTTP traffic sent
558,224,585 ms (155 hours) spent running SQL queries
99,346,916 ms (27 hours) spent on redis hits
132,384,059 ms (36 hours) spent on Tag Engine requests
2,728,177,045 ms (757 hours) spent processing in ASP.Net
(I should do a post on how we get those numbers quickly, and how justhaving those numbers is worth the effort)
Keep in mind these are for the entire Stack Exchange network but still donât include everything. With the exception of the 2 totals, these numbers are only from HTTP requests we log to look at performance. Also, whoa thatâs a lot of hours in a day, how do you do that? We like to call it magic, other people call it âmultiple servers with multi-core processorsâ â but weâll stick with magic. Hereâs what runs the Stack Exchange network in that data center:
We donât onlyrun the sites, The rest of those servers in the nearest rack are VMs and other infrastructure for auxiliary things not involved in serving the sites directly, like deployments, domain controllers, monitoring, ops database for sysadmin goodies, etc. Of that list above, 2 SQL servers were backups only untilvery recently â they are now used for read-only loads so we can keep on scaling without thinking about it for even longer (this mainly consists of the Stack Exchange API). Two of those web servers are for dev and meta, running very little traffic.
When you remove redundancy hereâs what Stack Exchangeneeds to run (while maintaining our current level of performance):
2 SQL servers (SO is on one, everything else on anotherâ¦.they could run on a single machine still having headroom though)
2 Web Servers (maybe 3, but I have faith in just 2)
1 Redis Server
1 Tag Engine server
1 elasticsearch server
1 Load balancer
(we really should test this one day by turning off equipment and seeing what the breaking point is)
Now there are a few VMs and such in the background to take care of other jobs, domain controllers, etc., but those are extremely lightweight and weâre focusing on Stack Overflow itself and what it takes to render all the pages at full speed. If you want a full apples to apples, throw a single VMware server in for all of those stragglers. So thatâs not a large number of machines, but the specs on those machines typically arenât available in the cloud, not at reasonable prices. Here are some quick âscale upâ server notes:
SQL servers have 384 GB of memory with 1.8TB of SSD storage
Redis servers have 96 GB of RAM
elastic search servers 196 GB of RAM
Tag engine servers have the fastest raw processors we can buy
Network cores have 10 Gb of bandwidthon each port
Web servers arenât that special at 32 GB and 2x quad core and 300 GB of SSD storage.
Servers that donât have 2x 10Gb (e.g. SQL) have 4x 1 Gb of network bandwidth
Is 20 Gb massive overkill? You bet your ass it is, the active SQL servers average around 100-200 Mb out of that 20 Gb pipe. However, things like backups, rebuilds, etc. can completely saturate it due to how much memory and SSD storage is present, so it does serve a purpose.
We currently have about 2 TB of SQL data (1.06 TB / 1.63 TB across 18 SSDs on the first cluster, 889 GB / 1.45 TB across 4 SSDs on the second cluster), so thatâs what weâd need on the cloud (hmmm, thereâs that word again). Keep in mind thatâs all SSD. The average write time on any of our databases is0 milliseconds, itâs not even at the unit we can measure because the storage handles it that well. With the database in memory and 2 levels of cache in front of it, Stack Overflow actually has a 40:60 read-write ratio. Yeah, you read that right, 60% of our database disk access is writes (you should know your read/write workload too). Thereâs also storage for each web server â 2x 320GB SSDs in a RAID 1. The elastic boxes need about 300 GB a piece and do perform much better on SSDs (we write/re-index very frequently).
Itâs worth noting we do have a SAN, an Equal Logic PS6110Xthatâs 24x900GB 10K SAS drives on a 2x 10Gb link (active/standby) to our core network. Itâs used exclusively for the VM servers as shared storage for high availability but does not really support hosting our websites. To put it another way, if the SAN died the sites would not even notice for a while (only the VM domain controllers are a factor).
Put it all together
Now, what does all that do? We want performance. Weneed performance.Performance is a feature, a very important feature to us. The main page loaded on all of our sites is the question page, affectionately known as Question/Show (itâs route name) internally. On November 18th, that page rendered in an average of 33 milliseconds. While we strive to maintain 50ms, we really try and shave every possible millisecond off your pageload experience. All of our developers are certifiably anal curmudgeons when it comes to performance, so that helps keep times low as well. Here are the other top hit pages on SO, average render time on the same 24 hour period as above:
Question/Show: 28 ms (29.7 million hits)
User Profiles: 39 ms (1.7 million hits)
Question List: 78 ms (1.1 million hits)
Home page: 65 ms (1 million hits) (thatâs very slow for us â Kevin Montrose will be fixing this perf soon: hereâs the main cause)
We have high visibility of what goes into our page loads by recording timings forevery single request to our network. You need some sort of metrics like this, otherwise what are you basing your decisions on?With those metrics handy, we can make easy to access, easy to read views like this:
After that the percentage of hits drops offdramatically, but if youâre curious about a specific page Iâm happy to post those numbers too. Iâm focusing on render time here because thatâs how long it takes our server to produce a webpage, the speed of transmission is an entirely different (though admittedly, very related) topic Iâll cover in the future.
Room to grow
Itâs definitely worth noting that these servers run atvery low utilization. Those web servers average between 5-15% CPU, 15.5 GB of RAM used and 20-40 Mb/s network traffic. The SQL servers average around 5-10% CPU, 365 GB of RAM used, and 100-200 Mb/s of network traffic. This affords us a few major things: general room to grow before we upgrade, headroom to stay online for when things go crazy (bad query, bad code, attacks, whatever it may be), and the ability to clock back on power if needed. Hereâs a view from Opserver of our web tier taken just now:
The primary reason the utilization is so low is efficient code. Thatâs not the topic of this post, but efficient code is critical to stretching your hardware further. Anything youâre doing that doesnât need doing costs more than not doing it, that continues to apply if itâs a subset of your code that could be more efficient. That cost comes in the form of: power consumption, hardware cost (since you need more/bigger servers), developers understanding something more complicated (to be fair, this can go both ways, efficient isnât necessarily simple) and likely a slower page render â meaning less users sticking around for another page loadâ¦or being less likely to come back.The cost of inefficient code can be higher than you think.
Now that we know how Stack Overflow performs on its current hardware, next time we can see why we donât run in the cloud.