Week 2 December 10 2006 Work Begins
Accompanying PhotosAt the beginning of the week I was wondering what I could possibly write about by the weeks end. My illness was dragging on, and it felt like I would not get any work done. At the same time I began to worry that I would not be well enough to start work when our new building was delivered late in the week. As it turns out I have heaps of ideas for things to write about. But most will be easier to understand if I explain a bit about the project I work on, and what I do. So the theme of this week will be work.
I originally started working at the South Pole on a projected called AMANDA (Antarctic Muon and Neutrino Detector Array). I had a contract for a winter over season to maintain the detector, which includes a significant amount of electronics and computers. On completing that contract I was offered a continuing position at the University of Wisconsin, Madison doing computing support for AMANDA, and helping with the preparation of a proposal for a new project to be called IceCube. With the successful funding of IceCube, and the merger of AMANDA into IceCube, I became an IceCube employee. This is the fifth year of IceCube, and third year of IceCube construction at the South Pole. We expect to finish building IceCube in 4 more year.
In a nut shell AMANDA and IceCube (I will refer to them as just IceCube from here on) are neutrino telescopes for conducting neutrino astronomy. I'll start by explaining the telescope part. For most people this will make them think of optical telescopes with lenses and mirrors. Some people might even think of radio telescopes with large parabolic dishes. Optical and radio telescopes have one thing in common, they look at the electromagnetic radiation spectrum. IceCube does not look in the electromagnetic spectrum, but rather detects particles, specifically muons. Neutrinos and muons are elementary particles smaller than atoms. Using particles instead of electromagnetic radiation is a very recent type of Astronomy, and when complete, IceCube will be one of the most advanced such telescopes in the world.
A neutrino is more difficult to explain. It is an elementary particle, of which there are 3 types (but I will refer to them as though there is only one) of neutrinos, and they have the amazing property of interacting very very poorly with anything. Scientists explain most of the universe using these elementary particles (of which there are more than just neutrinos) and forces. It happen that neutrinos interact using one of the forces which is much less likely to cause an interaction than the other forces. The result is amazing, for instance neutrinos produced in the sun can travel right through the earth and can continue on right across the universe!
If a neutrino interacts so poorly with anything, how do we detect them? While they interact poorly, every so often, very rarely, they do. When they do interact with something they produce another particle called a muon. A muon is a charged particle that does interact well with matter. So we look for muons. We detect muons by looking for Cerenkov Radiation, which they emit under certain circumstances. Cerenkov Radiation is just very faint blue light, which appears as a flash when produced by a muon. To see these faint flashes of blue light we need something for them to interact with and it ahs to be extremely clear and dark. The first neutrino detectors where built deep in mines or in the deep ocean or lakes, where the water is clear and dark. These detectors also need to be deep so that other particles do not reach the detector. The South Pole sits on a glacier made of ice almost 3km thick. Deep down below the South Pole the ice is extremely clear, and dark. Finally because this is such a rare event, you need to watch a large volume to see these particles reasonably often. This makes the South Pole perfect, the ice is dark, and clear, and there is lots of it. In IceCube we will watch one cubic kilometer of ice.
The final part of the puzzle is astronomy. For many years people just wanted to know more about neutrinos, and many people still study neutrinos. However IceCube's main goal is to use neutrinos to look out into the sky and see and understand things we have not been able to previously because of the limitations of astronomy using electromagnetic radiation. Scientists will use the results from IceCube to understand phenomena we have observed, but not well explained, such as Gamma Ray Bursts, or Super Nova, and hopefully to understand things we have not even observed, such as what is the dark matter - we know to exist in the universe, but have not observed it.
Part of the difficulty of explaining IceCube is the complexity. This complexity leads to a large project with many people working on it in many countries. I'll give a very high level overview for now. For a start we have to get our light detectors into the ice. To do this a hot water drill system was developed which can drill a hole at the South Pole through 2.5km of ice, producing a hole more than 0.5m in radius in less than 36 hours. The drilling itself is a huge part of the project, and the drill system had to be engineered specially for the project. Once the hole (which is full of water) is made a cable is lower into it, with detectors attached every 17m on the lower 1km. This process must be completed within 24 hours as the water in the hole starts to refreeze.
The detectors in the ice are photomultiplier tubes (PMT) which can detect single photons of light. They are inside a glass pressure vessel which can withstand the enormous pressures (10000 psi)of being 2.5km under water, and the additional pressure as the water re-freezes. Each PMT is controlled by a small computer, also inside the pressure vessel, which communicates with a computer in main laboratory, which can be over 3km away. Because of the computer and digital communications, these detectors are known as DOMS (Digital Optical Module Sensors). The design and engineering of the DOMS, and continual testing, is an enormous part of the project. Over 4800 DOMS will be manufactured in 3 locations around the world. They are extremely carefully designed and tested because once installed they can never be fixed. Over 15 years we expect less than 5% of the DOMS to fail.
Once the signal is received by computers at the surface, the signals from individual DOMS need correlating to find bunches of pulses associated with a particle going through the detector. Once these "events" are formed, analysis is done to determine the energy of the event and direction from which it came. With AMANDA we see about 100 events per second. In IceCube we expect to see 1500 events per second! Most of these events are muons or neutrinos produced in the upper atmosphere by cosmic rays. However every so often, about one in every 1000000, of the events is a particle which looks like it came from inside the earth. The only particles that can travel through the earth that we know of are neutrinos, and these are the events we are most interested in. We just have to sort through one million other events to find them! To do the things described above requires significant computing infrastructure, and very complex software.
This gets us to what I do in IceCube. I provide computing infrastructure for the project, of which the surface computing at the South Pole is large part. I lead a team of about a dozen very experienced and skilled people who really make this possible. About half of them will work at the Pole this year. While I like to think I am still technically competent, I do less computer administration, and spend more time on budgets, schedules, meeting, and reviews every year.
Until this year the computing at Pole has been housed in a temporary building not very suited for operating a data center. This year we finally get to move into our new data center. To do this we built up a completely new system, which we crated up and sent south about 2 months ago. On Saturday we were finally given access to our new building and we have been working fairly solidly to get equipment in and set up since.
All fairly straight forward stuff until you add the "Antarctic Factor". The new building we moved into was originally used for berthing, the old El Dorm (Elevated Dormitory). Two years ago it was relocated to the center of IceCube and work begun to refinish for our needs. It was scheduled to be delivered last year, but the schedule slipped. I have been working on aspects of the design for the data center since beginning on IceCube, maybe 5 years ago. Even still it was looking like a close call as to whether we would get access by the scheduled date this year, which was extremely important because the enormous amount of work we have to do in it this season. The final hurdle was an inspection by a design team before the government would take ownership from the contractor, and we could use it. During the inspection it was suddenly discovered that the way I had planned to space the racks did not provide for two clear straight 36 inch egress paths from the most isolated point in the room. Why this was not discovered long ago, since this drawing has been near final for over 2 years, I have no idea. Also 3 support pillars were much larger in real life than on the architectural drawings. One of the most important racks, were all the network trunk lines originate, had to be moved all the way to the other end of the room. We pre built this system in Madison, mocking up the room to exactly what we expected to have delivered. All the cables were custom made to length, and carefully labeled and recorded. Suddenly on the first afternoon I was feeling well enough to work I found myself significantly redesigning the rack layout of our data center, with no clear idea on how we would deal with the changes.
The data center is interesting in that along with the computing there are other considerations, such as bringing in the huge surface cables from the ice, and connecting them to the controller computer. some engineers were concerned about potential Electromagnetic Interference (EMI) problems, and the data center became a Faraday cage. Then at the eleventh hour it was decided there needed to be fire suppression. I still don't understand if it is designed to protect from a fire inside the data center or in the rest of the building. The result is a data center that isn't exactly what I had in mind. It reminded me of the joke "What is a camel? A horse designed by committee.". This is certainly the camel version of a data center.
While we were having fun inside, the inspectors had discovered that a few of the external emergency stairs were off by rise, or depth by as much as 1/16 of an inch! Along with a few other things like a wall having to be shortened by 3 feet resulted in our new building, the ICL (IceCube Laboratory) looking like a forced labor camp! At the final moment the inspectors passed the building and we were given Conditional Occupancy.
On Saturday we had a crane lift over 40 crates of equipment (about 20000 lbs) to the data center, resulting in a extremely packed room. We spent a good part of today unpacking UPSs (Uninterpretable Power Supplies) and computers, and installing them into racks. We even got the racks into their final locations. We now have enough space for the remaining equipment to be lifted up tomorrow. If we just had power we would be set!
During the summer the South Pole is all about work. There isn't a huge amount of time or energy left over for recreation. This has changed a bit in the time I've been coming here. There has always been competitions and bands and parties. But these days there is someone who officially organizes these activities. It feels a bit manufactured and a bit forced to me. I guess for the new people coming through this will be their traditions, and their experience of Pole. But it is not mine. I do like live music, and usually go to see any band. But Saturday I was tired and didn't feel like standing around in Carharts at a Toga party. If I'd had the energy I would rather have spent the night playing cribbage with mates in the bar. Maybe next week.
To be continued....