Kristian's web log

February 17, 2010

Wikish – Mediawiki-editing on command line

Filed under: /dev/random — Tags: , , , , — kristian @ 00:09

wiki.sh (or wikish) is a shell-script I wrote to work with our internal mediawiki-wikis. It’s functional for both the new varnish-software wiki and the old redpill-linpro wiki I use.

Basically it lets you edit wiki-content with vi, vim, gvim or even nvi. Very simple, and a result of mvs, wikipediafs and mediawiki-fuse being seemingly horrible messes. It also makes it possible for me to easily script copy/pasting from one wiki to an other without much hassel. All in all it’s a little over 200 lines of sh-code using lwp-request/curl and awk. It works.

Get it from: http://github.com/KristianLyng/wikish

Currently doesn’t support “normal” mediawiki-login, but rpl and varnish-software use basic auth over ssl and work fine. I’ll gladly add normal login if someone gives me an account on a wiki they own and operate where I can test it (not wikipedia please, unless you are wikimedia).

Usage example: wikiedit “Main Page”. (that simple, yes)

PS: Sorry for hijacking the name.

January 19, 2010

Real-time and hit-and-run based statistics for VSTS

Filed under: Varnish — Tags: , , , , — kristian @ 14:18

A while ago I wrote VSTS – the Varnish Stress Testing Suite. It’s not nearly as fancy as the name makes it sound: It’s a simple set of shell scripts that runs on a set of test servers and pounds Varnish under a couple of different scenarios. The idea is simple: Detect possible performance issues during the development cycle of Varnish by periodically testing the current code against the same well-established test.

It does the job, but could be far better. Specially when it comes to statistics.

The rrdtool-backed Python script is simply insufficient. As rrdtool likes to put data entry into a time-slot, and I don’t really operate in time slots, the data is both slow and imprecise. This is fine if you want data for every 5 minute throughout a year, but not if you want data collected anwhere from one time to ten times a day. Simply put: I want one data entry to represent one unit on the X-scale.

I also want to add “real-time” statistics. The current data is collected after each test, but I want data to be gathered throughout the relevant tests. This should be seperate from the other statistics. Hopefully, the system I end up with is flexible enough to allow me to plot all the datasets on the same graph, which should make any variations easy to spot.

Up until now I’ve only ever dealt with rrdtool when it comes to statistics (unless you count pre-rrdtool mrtg and the like), so I was hoping for some pointers before I start digging through what I suspect is a jungle of similar but different solutions. I’m comfortable working with most languages, and the primary concerns I’m looking at is implementation complexity and flexibility.

When I’m done with this batch of refactoring, I’ll do a proper writeup. However, what I’m most excited about is adding OpenSolaris into the target-platforms (should help clean out a few bugs that have been plaguing the Solaris-users for a while), adding better support for customized tests and improved robustness of VSTS.

August 24, 2009

Saint Mode

Filed under: Varnish — Tags: , , — kristian @ 16:45

SSIA?

Saint mode is now committed to trunk. This means you can evaluate responses in vcl_fetch and choose to discard them, hold off requests to the backend for N seconds and use graced objects instead. In other words, if your image-site is delivering images with http 200 OK but a single-digit content-length, you can decide to not use that response, but used the previously cached one instead (and/or try a different backend).

The syntax is reasonable simple. We’ve introduced the ‘beresp.saintmode’ variable which can be set in vcl_fetch. This adds the objecthead to a list of ‘troubled’ objects hanging off each backend. This means that if one backend is broken but an other is fine, only the broken one will be blacklisted for that specific object. After this is set, you need to restart. So something like this:

if (beresp.status == 500) { set beresp.saintmode = 20s; restart; }

Unfortunately we ran into a snag with regards to doing this in vcl_error, since we don’t have what we need there. This is more of a general problem than saint mode-specific, and with PHK’s blessings, I’ll be writing a workaround/fix for this with regards to saintmode tomorrow. It should be reasonably simple to fix, if the VCL syntax is acceptable. (We’re talking an hour or two + testing).

July 29, 2009

Saint mode?

Filed under: Varnish — Tags: , , — kristian @ 16:08

Yet another implementation of saint mode part 3. With some luck, we can do this in trunk soon[tm]. The idea is to let VCL decide, based on the backend response, whether to used a cached object or a fresh one.

This is controlled in VCL ( if(beresp.http.foo == Yes) { set beresp.holdoff = 20s; } ). Not quite ready, but fun nevertheless.

16:56:09 200 OK Age: 0
16:56:10 200 OK Age: 1
16:56:11 200 OK Age: 2
16:56:13 200 OK Age: 4
16:56:14 200 OK Age: 5
16:56:15 200 OK Age: 0
16:56:16 200 OK Age: 1
16:56:17 200 OK Age: 2
16:56:19 200 OK Age: 4
16:56:20 200 OK Age: 5
16:56:21 200 OK Age: 0
16:56:22 200 OK Age: 1
16:56:23 200 OK Age: 2
16:56:25 200 OK Age: 4
16:56:26 200 OK Age: 5
16:56:27 200 OK Age: 0
16:56:28 200 OK Age: 1
16:56:29 200 OK Age: 2
16:56:31 200 OK Age: 4
16:56:32 200 OK Age: 5
16:56:33 200 OK Age: 0
16:56:34 200 OK Age: 1
16:56:35 200 OK Age: 2 <—- Set unwanted header on backend response
16:56:37 200 OK Age: 4
16:56:38 200 OK Age: 5
16:56:39 200 OK Age: 6 <— ’saint mode’ is in effect.
16:56:40 200 OK Age: 7
16:56:42 200 OK Age: 9
16:56:43 200 OK Age: 10
16:56:44 200 OK Age: 11
16:56:45 200 OK Age: 12
16:56:46 200 OK Age: 13
16:56:48 200 OK Age: 14
16:56:49 200 OK Age: 16
16:56:50 200 OK Age: 17
16:56:51 200 OK Age: 18
16:56:52 200 OK Age: 19
16:56:54 200 OK Age: 20 <— Return backend to normal.
16:56:55 200 OK Age: 22
16:56:56 200 OK Age: 23
16:56:57 200 OK Age: 24
16:56:58 200 OK Age: 25
16:57:00 200 OK Age: 0 <— Varnish catches up.
16:57:01 200 OK Age: 1
16:57:02 200 OK Age: 2
16:57:03 200 OK Age: 4
16:57:05 200 OK Age: 5
16:57:06 200 OK Age: 0
16:57:07 200 OK Age: 1
16:57:08 200 OK Age: 2
16:57:09 200 OK Age: 4

Btw, this is the output of:
$ while sleep 1; do FOO=`date +%H:%M:%S`; echo -n "$FOO "; GET -Use http://localhost/~kristian/index.cgi | egrep '(^Age|^200|^503)' | perl -e 'while () { chomp; print; print " " }; print "\n"'; done

July 5, 2009

Pop Quiz!

Filed under: /dev/random — Tags: , — kristian @ 22:17

I like funny little problems – but more importantly: the different ways people solve them. Hope you do too. I’ve only got two questions this time around.

 Question 1: How do you delete a file named -foo ?

Ok, sorry, this is a mandatory question. It ALWAYS comes up. The well-documented answers are: rm — -foo and rm ./-foo. Do you have any other simple answers?

Question 2: How do you easily list all directories in a directory with both files and directories?

The easiest way I’ve found is «ls -d */», but you could always fall back to find («find -maxdepth 1 -type d»), but that’s boring. Piping and grepping on ls -l was suggested too…. Any other interesting solutions that I’m missing?

March 31, 2009

Varnish, Munin, RRDTool and RPN – Yaaai

Filed under: /dev/random, Varnish — Tags: , , , — kristian @ 22:06

I’ve spent a couple of days working on a new munin-plugin for Varnish, and it’s been interesting. First of all, there are several existing Varnish plugins, all slightly different with their own quirks. This new one will replace the plugin in the varnish subversion repo and hopefully enter the list of official munin plugins.

Spent monday setting up the basic plugin, and all day (tuesday) trying to figure out how the heck to display hitrates sanely. The statistics varnish delivers are ever increasing, which munin handles exceptionally well (well, rrdtool does). The only way to display it relatively (in percentages) is to either store history on the node (which is out of the question) or fix it up with a CDEF.

As it turns out, there’s a slight bug in Munin trunk with regards to the CDEF-solution, in addition to a lack of working examples of how to do this. Nicolai helped me track down the Munin-issue (patch comming), so then it was only the RPN-insanity…

'cdef' => 'client_req,0,EQ,0,cache_miss,client_req,/,100,*,IF'

This little beauty is obviously the equivalent of

if (client_req == 0) {
0
} else {
(cache_miss / client_req) * 100
}

That is, if I actually got that right. So in this case, it defaults to 0 cache misses if there are no requests in the relevant period.

Right now I’m at the point where the graphs _draw_ correctly but the numbers just don’t add up. This only applies to the percentage-based graphs, which so far is only the hit_rate graph. I guess that’s a problem for tomorrow.

At any rate, I’ll see if I can’t get some real-life data later this week, in the meanwhile, here’s the four first graphs with random httperf-generated data…

Request rate Threads
Backend traffic Hit rate in percent

March 28, 2009

VDPAU – 1080p with CPU cycles to spare

Filed under: /dev/random — Tags: , , , — kristian @ 20:00

I just went from Ubuntu 8.04 to 9.04 (through 8.10). For two reasons:

1. EXT4

2. VDPAU

In Ubuntu 9.04, the 180-series of drivers from nVidia is included. Doesn’t matter much? Well, it does have VDPAU. That means GPU-assisted decoding of h264. I’m currently playing a 1080p video on my Core 2 Duo 2.4GHz on battery power and it’s smooth. Very smooth.


kristian@kjeks:~$ while sleep 1; do cat /proc/cpuinfo | grep MHz; done
cpu MHz : 2000.000
cpu MHz : 800.000
cpu MHz : 2400.000
cpu MHz : 800.000
cpu MHz : 1600.000
cpu MHz : 800.000
cpu MHz : 1600.000
cpu MHz : 800.000
cpu MHz : 2000.000
cpu MHz : 800.000
cpu MHz : 2000.000
cpu MHz : 800.000

That’s my cpu clocking down to conserve battery and because the decoding isn’t taking place on the CPU any more. Mind you, the CPU usage is still pretty high, but the difference between 70-90% of CPU usage and 100% is the difference with decoding just in time and decoding just a bit too late.

One side note is that VDPAU doesn’t work while running Compiz. Yet.

To get this madness to work, nVidia has made a nice little script and patch to compile mplayer with the required support. Available from ftp://download.nvidia.com/XFree86/vdpau/, it’s surprisingly clean. These things have a nasty habit of doing Bad Stuff, but this really is as clean as it gets. It’s essentially just a few svn checkouts followed by patch -p0, ./configure and make. Not even make install. Oh yeah, if you’re doing this you obviously need patch, subversion and the ability to build mplayer.

After that, all you do is grab your favorite h264 1080p video and play it with mplayer -vo vdpau.

February 22, 2009

Playing with swap on Linux

Filed under: /dev/random — Tags: , , , — kristian @ 04:12

Swap is a funny thing.

I’ve got 4GB of memory on my desktop at home, and I’ve been experimenting lately with different swap setups. For the longest time, I didn’t really run any swap on desktop setups with a reasonable  amount of memory, but once I installed this one, I went with the default 1.5GB that Ubuntu gave me.

Most of the time, it’s not in use. However, I ran in to a funny bug with MPlayer which, combined with my 800MB cache, resulted in 3×800MB mplayers lingering in the background. When you add Firefox and three virtual machines into the mix, you’ve got yourself a swapping machine.

So I got curious. I know that with my setup, most of the memory usage is inactive. We’re probably talking about Firefox history/cache, virtual machines with free memory and so on, so I figure, why not use it for disk cache instead? Well, I tried. I happen to have two 37GB 10k raptors in my machine. One of them wasn’t doing anything so I set it up as swap.

Now, to control swap-behavior under Linux, you use /proc/sys/vm/swappiness. There’s been some debate as to what is a sane number, but it ranges from 0 to 100. Default was 60 on my Ubuntu machine. I first tried setting it to 100, causing maximum swapping. The result was nothing short of nasty. After I’d left the machine alone for a while (for half a day, for instance), it would seem that just about everything was swapped out, and I’d get nasty delays just opening yet an other terminal or even closing one. We’re talking seconds. For everything. Clearly, this was not very nice.

To make a long story short, I discovered that Linux doesn’t seem to be as good as I hoped. Even when I played large amounts of video, the disk cache didn’t seem to go up, yet things were still getting cached. In fact, most of my memory seemed to be use for swap cache. Turning the swappiness down to 70 seemed to do little good, though it did improve the snappiness of the desktop considerably. However, I was not happy with the results, so I tried to turn off swap.

That’s when the fun begun.

Apparently, Linux doesn’t like it when you rip out 5GB of virtual memory that’s in use.

My system essentially froze for 10 minutes while swapoff was running, and only woke up when oomkiller woke up to kill swapoff. To get my swap turned off, I first had to turn off firefox… Amazing.

At any rate, I’ve turned the swap back on and I’ll be testing with swappiness of 30 for a while. Hopefully, that will make more sense.

Here are some possibly interesting munin graphs from the testing period:

Day

Week

Month

February 3, 2009

1080p under GNU/Linux

Filed under: /dev/random — Tags: , , , — kristian @ 19:15

I recently bought a 24″ monitor with 1920×1200 resolution, and suddenly found myself actually caring about HD. I assumed my Core 2 Duo E6600 (2.4GHz) combined with nVidia 8600GT would handle it…

Now, as it turns out, it ALMOST handles it. By renicing mplayer to -19, I can mostly get through the munin update that polls localhost + 3 virtual machines running on localhost without mplayer lagging. The framerate might take a bit of a dip, but I can live with that (after all, that’s a significant load spike).

For those who aren’t familiar with HD, 1080p refers to video with a vertical resolution of 1080 pixels, typically 1920×1080. The p indicates progressive scan, as opposed to interlaced. The codec used is typically H264, which is what I’ll focus on.

Anywa… I just discovered that nVidia released a driver as late as January 8th 2009 that contains support for GPU-accelerated h264 decoding. They even supplied ffmpeg and mplayer patches to enable it. As much as I dislike the proprietary aspect of nVidia, I must say that they have always been miles ahead of everyone else with their Linux drivers. This nvnews article announces the support for VDPAU, the API needed to do all this magic. It also points out that the support is preliminary, and that it’s not stable yet. But for any HD-loving GNU/Linux-user with an nVidia card, this is great news.

Back to my non-gpu-accelerated setup… While it currently works, I won’t say it works great. The cpu core with mplayer is resting neatly at 95-100% usage, and that’s not good. Additionally, I’m getting quite a bit of tearing, which I’ll look into. Hopefully, I can solve that with some vsync-tweaking… I am, after all, running compiz during all of this.

Powered by WordPress