Life and all that

I’m rubbish at keeping a blog I have decided, I have things I want to talk about in my head but none which I want random people to read. Pretty sure that’s what blogs are for. Maybe it’s because I’m not an angst ridden emotional teenager and I want to keep my private life to myself.

Anyway, what have I been doing. World of Warcraft wise, big success. We finally cleared Ulduar (ignoring the horrible hard mode boss) and have moved on to Icecrown Citadel. We’re already 2 bosses through, the prospect of clearing all the raids in Wrath before Cataclysm is released is looking surprisingly likely! I’m really pleased for our little crew, from such small beginnings, etc. It’s also nice to be killing bosses who drop good loot as well, I’d forgotten what that felt like. Helps to gear up some of our more reluctant raiders. Oh, also invited some new people to the fray, a level 80 rogue and a level 13 paladin – friends of friends, both at Lancaster Uni. I like making new friends, so fingers crossed they’ll fit in well

Real life time now, I’ve got so many exciting things coming up this year I can’t quite contain myself. It starts on the 3rd of March when my brand new 27″ iMac is delivered. Then on the 12th March the F1 season starts (yes I am UNBELIEVABLY excited about that), then the 15th of March I am FLYING TO CALIFORNIA to visit Matt and Lea. Yes you read that right, I am finally making the trip to the states I have always wanted to make, with the added benefit of seeing good friends when I am out there. Then I come home, and I have booked my Motorbike training for the 3rd April, and then  I might be going to Paris (conference submission pending). Then in May we’re making plans to possibly go to Turkey for the F1 Grand Prix!!!!! And in June, Heather wants me to go on a cheap holiday with her too!! My passport will be getting a real work out.

I have a feeling this is all going to make me very poor but I just can’t bring myself to care.

Been thinking a bit about what I might do when I finish my PhD (I know, putting the cart before the horse somewhat given I’m still frantically working on PhD stuff, but humour me). I’m not sure I am cut out for a career in research, well not in the sense of sitting in Infolab working on specific projects, I try but I am learning my brain just isn’t in it. I feel a bit like I want to get out and see the world – I guess this is still some hangover from not having a future anchored to employment at the university any more, and probably why I am going traveling/experience mental in 2010. Of course my ideal employment would be something in the glamorous world of Formula 1, but I doubt I am mathematically clever enough to even dream of a technical job, and everything else is being in the right place at the right time, and knowing the right people. So I’m back to what was my original idea as a kid (other than joining the RAF, which I guess isn’t off the cards either… I just don’t think I ever could bring myself to be physically fit enough these days ), something in the Foreign and Commonwealth office, I can be diplomatic right? I’m being purposefully vague, I know more about this than I’ll bore people with, but there’s a few possible avenues in that direction I could explore, and I guess traveling, studying, and meeting new people are all useful experience for such a career.

Been feeling a bit weird recently, sort of homesick for my parents’ house. I think I just miss having someone look after me. I have lots of friends in Lancaster of course, and beyond, but there’s nothing quite like being looked after, and fed, and just being able to recline in the living room watching crap on TV. So what I’m saying is I want a servant. No, wait that’s not right… Seriously though, I went home for the weekend and it was really nice, lovely warm house, very relaxing. Of course parents are parents, they get on your nerves a bit, but I reckon with a real internet connection staying there would be really good. As long as I had my desk and everything else of mine down there. I think really its just like being in a bubble, nothing to worry about, forgetting about the real world, just letting it pass you by. And I can’t really do that, I have things to do and a PhD to finish!! Makes me wonder what I’ll do when I reach the end of my funding though, come November. I think its likely I’d just move back there for a buffer, bit of breathing room. We shall see.

Anyway, lets end on a less introspective and boring note. Television! I have regained my appetite for television. Firstly, I am becoming something of a Doctor Who nerd, of the highest order. I’m ridiculously excited about the new series starting soon, and I’ve been re-watching loads of the 2005+ stuff. I’ve downloaded some original Who, but I am struggling to work out where to start watching it all. It’s complicatedly ordered and impossible to order which upsets my digital OCD. Other than that I watched the new series of Silent Witness on the BBC and it was amazing. Literally amazing. I recommend people go watch it, I’m sad it has ended, and on such a whimper really too. I didn’t even know it had ended, got excited when I saw there was more on TV and then realised it was them re showing the previous season. Which I haven’t seen so that’s sort of good but… Lastly I’ve finally got around to watching Life on Mars (series 1 so far so no one spoil the end for me), and that is amazing too!!!!! So many TV shows, so little time.

Will end here, I have rambled enough. Removes my guilt for untimely updating for a while at least.

I’ve recently been working on a personal project to set myself up a fileserver with redundant storage, and I thought I’d combine this with a desire to run my own installation of Myth TV. In the house I’m currently living in, my room is in the attic and the only aerial input is fed into the living room, Myth seemed like the perfect way of distributing the television upstairs without cables, and theoretically should plug directly into the XBMC install I have running on an Asus Revo R3600 I have upstairs.

So this is detailing my experiences, the problems I’ve come up against, and the potentially rubbish ways I’ve come up with to deal with them. The end result however, is something that appears to be working, and whilst I can’t vouch for the success of my redundant storage solution till I have to use it in anger, it seems to be functioning as expected.

Step Zero: Mythbuntu

I chose to go with the pre-built Myth distribution of Ubuntu. I presume that most of these instructions would work for any debian based flavour of Linux (and with a bit of thought probably any other distro) but I didn’t have to manually install any of the myth software, and my hardware was all auto detected. I suppose in some cases you wont be so lucky.

One small issue I think I came across, I installed the OS and then found that GRUB couldn’t find the master boot record and it refused to boot. I think this was because the system disk wasn’t the first one seen by the BIOS. I just switched my SATA connectors around until it was, and reinstalled from scratch. Not the perfect answer, but it worked and that was my goal

Step One: Software Raid

I used mdadm to handle the RAID, and this wasn’t pre installed in Mythbuntu so first of all, get it installed.
$ apt-get install mdadm

In my machine I started off with two 1.5Tb hard disks, alongside the system disk, which I wanted to set up as a paired raid10 array. I searched dmesg to find the device names of the two disks first.
$ dmesg | less

Piped the output of dmesg into less, then searched that output for references to the one disk I knew I had, the system disk ’sda’.
'/sda'

This showed me sdb and sdc, which I guess you’d think was logical, but I was being paranoid. You can also see the devices listed in /dev/.

Here was my magical command :-
$ mdadm -v --create /dev/md1 --level=raid10 --raid-devices=2 /dev/sdb /dev/sdc

Verbosely create a new raid 10 device called /dev/md1 (I should have used md0 if I were being correct but I didn’t, meh), using two raid devices found at /dev/sdb and /dev/sdc. You can test this is working with :-
$ mdadm --detail /dev/md1

And again, the device should appear in /dev/.
You can check the progress of the raid array being initialised with :-
$ cat /proc/mdstat

And you can see the raid array appearing as one disk, by using :-
$ cfdisk / dev/md1

Though this may not work as expected if the disks you’ve used already had some form of file system on them, unreadable by Linux.

Technically there is another step which goes here, but I’m writing this to include the problems I faced, as I faced them; skip down to ‘Problems after Reboot’ if you want to spoil the surprise.

Step Two: LVM

Initialise the disk for LVM.
$ pvcreate /dev/md1

Creates the a volume group, containing that ‘disk’.
$ vgcreate vg-server1 /dev/md1

Now, create the volumes themselves! I decided I wanted two, one partition for mythtv, the other for data.
$ lvcreate -L 500g -n lv-mythrecordings vg-server1
$ lvcreate -L 895g -n lv-data vg-server1
I had a ‘1.5Tb’ disk, which of course ends up being nowhere near 1500gb. Hence I chose 500gb for my myth partition, and what was left at this point for data.

Next I needed to put a filesystem on the partitions. I chose XFS as it is extendable while mounted, very easily, but it has the drawback of not supporting shrinking. I don’t think I’m ever going to need to make my partitions smallers, so that’s fine by me.
$ apt-get install xfsprogs
$ mkfs.xfs /dev/vg-server1/lv-data
$ mkfs.xfs /dev/vg-server1/lv-mythrecordings

In order to test this had worked I mounted the partitions, and wrote small text files to them.
First of all, create the mountpoints (I did this in /mnt for convention purposes at this point) :-
$ mkdir /mnt/data
$ mkdir /mnt/mythrecordings
$ mount /dev/vg-server1/lv_data /mnt/data
$ mount /dev/vg-server1/lv_mythrecordings /mnt/mythrecordings
When mounting you need to refer to the ‘magical’ device address, you can see how it works there.
Also, if you mount these as root (i.e. sudo) you can only write to them as root. Expect to have to sudo when creating your test files.

Step Three: fstab
We don’t want to have to mount the partitions by hand every time, it’d be ridiculous. The solution is to add them to fstab.

My editor of choice is vim, but any text editor will do :-
$ vim /etc/fstab
You may wish to make a copy before you edit this file, just in case you change something by accident.
I added :-
/dev/vg-server1/lv-data /raid/data xfs defaults 0 0
/dev/vg-server1/lv-mythrecordings /raid/mythrecordings xfs defaults 0 0

The ‘defaults’ option means certain interesting things to do with the way in which the partitions are mounted. The default was fine by me, read up on it if you think you may need something more complex.

Now create folders for the mount points to match fstab. I chose to keep my partitions mounted in /raid – something I just chose myself, no convention.
$ mkdir /raid/data
$ mkdir /raid/mythrecordings

I then tested fstab by mounting them as they should be when the system boots :-
$ mount /raid/data
$ mount /raid/mythrecordings

When mounted, you will need to check the permissions on the folders are correct for everything to have permission to read/write to them, and that they are owned by the right user / in the right group. This involves the commands ‘chmod’ and ‘chown’.

Step four: Samba

Now samba for remote access to the partitions as shares, from Mac OS and Windows (and Linux).

$ apt-get install samba smbfs

Then you just add things to the config file. Again, make a copy if you’re worried you might mess it up.
$ vim /etc/samba/smb.conf

Under global settings I put :-
security = user

And then I added my shares :-
[media]
comment = Media Share
path = /raid/data/media
browseable = yes
read only = yes
guest ok = yes
write list = sara

[unsorted]
comment = the unsorted directory
path = /raid/data/media/Unsorted
browseable = yes
writeable = yes
guest ok = yes

[files]
comment = Files Share
path = /raid/data/files
browseable = no
read only = yes
guest ok = no
read list = sara
write list = sara

[sab]
comment = sabnzb space
path = /raid/data/sab
browseable = no
read only = yes
guest ok = no
read list = sara
write list = sara
These may not be exactly right, but they seem to work for me. Notably, [unsorted] is a dump directory. Guest is allowed to write to it, to allow housemates who I don’t trust to organise things properly to leave things for me to assimilate into the media collection. [media] is readonly for guest accounts (i.e. they are allowed to read it, and access what’s there, but they can’t mess anything up). [files] and [sab] are shares only for me, they don’t appear in a network browser and only I have access to write.

Lastly, whichever account you’ve specified as having access, you need to give it a samba password.
$ smbpasswd -a sara
New SMB password:
Retype new SMB password:
Added user sara.
This only works for user accounts which exist on the host machine.

Step five: Problems after Reboot

Having got this far, I turned off the machine to connect my second pair of hard disks. On booting the machine up, the raid device /dev/md1 was nowhere to be seen but I could see that /dev/sdb and /dev/sdc still existed.

I tried to manually start the raid array first of all :-
$ sudo mdadm -A /dev/md1 /dev/sdb /dev/sdc
[sudo] password for sara:
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: /dev/sdb has no superblock - assembly aborted
Obviously wasn’t successful.

So, I wanted to check that the disks still knew they were meant to be part of an array :-
$ sudo mdadm --query /dev/sdb
/dev/sdb: is not an md array
/dev/sdb: device 0 in 2 device undetected raid10 /dev/md1. Use mdadm --examine for more detail.
$ sudo mdadm --query /dev/sdc
/dev/sdc: is not an md array
/dev/sdc: device 1 in 2 device undetected raid10 /dev/.tmp.md1. Use mdadm --examine for more detail.
This made me breathe a little easier.

In the end, the solution was to create a mdadm config file. And it turns out it comes with a nice little script to generate just that :-
$ /usr/share/mdadm/mkconf>/etc/mdadm/mdadm.conf

A reboot showed this had worked quite nicely.

Step Six: Adding more disks

I connected up my new disks, and I checked again in dmesg that they were being seen. I then double checked using mdadm that the disks i was about to raid, weren’t the previous disks being detected in a different order.
$ mdadm --query /dev/sdb
/dev/sdb: is not an md array
/dev/sdb: device 0 in 2 device active raid10 /dev/md1. Use mdadm --examine for more detail.
$ sudo mdadm --query /dev/sdc
/dev/sdc: is not an md array
/dev/sdc: device 1 in 2 device active raid10 /dev/md1. Use mdadm --examine for more detail.
$ sudo mdadm --query /dev/sdd
/dev/sdd: is not an md array
$ sudo mdadm --query /dev/sde
/dev/sde: is not an md array

This quelled my paranoia, so I went on to create the array :-
$ mdadm -v --create /dev/md2 --level=raid10 --raid-devices=2 /dev/sdd /dev/sde

And as before, add it to lvm :-
$ pvcreate /dev/md2
$ vgextend vg-server1 /dev/md2
This extends the volume group from earlier to now include this new hardware.

I unmounted the current shares I want to extend:-
$ umount /raid/data
Incidentally, smbstatus will tell you who/where is connected to the samba shares if you need to know.

$ lvextend /dev/vg-server1/lv-data /dev/md2
This extends the volume itself to include the maximum space available on the new disk.

Now, to grow the filesystem. The scary bit. XFS can only grow when mounted, so I remounted the share. Then grow the fs by referring to the mount point.
$ xfs_growfs /raid/data

Phew. I then ran lvscan to look at my volumes and their sizes :-
$ lvscan
ACTIVE '/dev/vg-server1/lv-mythrecordings' [500.00 GB] inherit
ACTIVE '/dev/vg-server1/lv-data' [2.24 TB] inherit
Success!

Step seven: A few tips on making myth frontends work with a remote backend

You need to make sure that myth-backend is set to refer to it’s external address, not localhost or 127.0.0.1. This is because when frontends connect, the backend tells them that address to talk to for video.
Also, don’t be foolish enough to try using MDNS (i.e Bonjour) for this. I had my external address set to ‘computername.local’ which works perfectly for every communication I would normally do. Unfortunately, it caused the official myth-frontend for OSX to fail to connect to the backend 9 times out of 10.

I found XBMC to be much, much nicer and more sensible. In order to add myth to it, you need to create a new source like this
myth://mythtv:YOURPASSWORD@mythbackend-pc
Make sure the password is the one from the myth-backend config, it is randomly generated.

Lastly, I got XBMC connected, it would show me the programme guide, and see a list of recorded programmes, as well as a list of live channels. Everything looked great, but when I tried to play any video it would either do nothing, or give me an error telling me ‘playback failed’ and asking me to check the logs.

Solution? Edit /etc/hosts, and add the fileserver by hand. Madness.

The End
I hope maybe this will help someone out there, it isn’t a perfect method and I’m sure I’ve done things in ways which could be improved, but it works and I am happy with it. I’d suggest if anyone has questions, I’m not the first person to turn to, I’ll only google for the answers!

I stayed up really late again last night, because I was worried about a meeting today at midday, and I’d broken something I was meant to be demonstrating. As a result of staying up late, I slept through my meeting. I just wanted to emphasise the idiocy there. I was woken by a phone call from my supervisor, which I was too scared to answer (and too incoherent), I had to ring him back a few minutes later apologising for being an idiot.

I guess that’s some kind of irony, missing the meeting you were worrying about because you were worrying about it. Anyway, turned out to be okay, I had a separate meeting later in the day, sadly missing my second supervisor, but we’ve arranged some things for me to be doing over Christmas. Everything feels better now. I had some more fail when I got into the office, pouring orangina all over my desk, laptop and monitor. I now have a sticky escape key. Thankfully the adage of bad luck coming in threes doesn’t seem to have held true.. yet. I’ve just spent the rest of the day with a lack of sleep headache, which I should be rectifying now by sleeping, but I felt like writing something. Blogging is kind of addictive when you get into it.

Watched ‘Taken’ this evening, film about Liam Neeson’s daughter being kidnapped by people traffickers. It was.. alright I suppose. Entertaining enough, but the premise didn’t seem to make sense. I understand he’s seen a lot of terrible things in the world, but kids go on holiday all the time, he didn’t need to have an irrational fear of his daughter going to Paris. And then the horrible something that did happen seemed so unlikely, why would they get picked up by people traffickers from the airport, why would people traffickers take obviously rich and well connected young Americans? Surely that’s just asking for trouble. Maybe I’m over thinking this…

It did make me want to watch Love Actually though (Liam Neeson is in it..), I know it makes me a big ol’ girl, but I really do love that film. I actually paid real money for the soundtrack yesterday, as I realised I’d lost my copy, it makes me feel happy for some reason, not entirely sure why. Doesn’t evoke any particular memories. Which is probably a good thing at the moment to be honest.

Tomorrow is the Christmas Meal for Computing and I’m looking forward to it, me and Faisal are going to watch the lecturers in their natural habitat, propping up the bar. More than anything I am excited about the guild meetup on Saturday. I hope there is enough room for everyone in the house, Friday is going to be a day of shopping and tidying up I feel.

I’ve spent a lot of time today and yesterday playing around with web software, notably Wordpress (which has got so much nicer to install and use since the last time I was poking it) and today Gallery3. I’ve had a gallery online for quite a long time – it  sits alongside a blog I try to write when I upload anything interesting. Since I’ve been moving hosting it gave me an excuse to have a rethink about how I’ve got it all set up.

Step one was moving to Gallery3, it’s still in beta but at the final stage of beta testing so I decided I’d make the leap. The install process is a lot smoother than previous versions, inspired I’d guess by Wordpress. All in all it quite simple, and I found that the directory structure Gallery2 had used meant it’s been very simple to import all of my old albums. the only issue I’m noticing so far is a lack of decent themes, and some issues with the javascript configuration menus. Some of that could be the browser I’m using (Google Chrome) however. I’m using the ‘darkness‘ theme, and it’s not bad, in fact if the menu issues are Gallery rather than the theme I’d say it’s pretty damn fine, but I think once I’ve finished adding my content I’ll be making some manual tweaks to the CSS.

Other than some teething trouble it’s been a fairly pain free process, which is very different from my previous memories of getting Gallery to work. Seems like a well written, and timely update.