Monthly Archives: July 2007

Shout Out

The first employee that I hired at my current job left for Silicon Valley on Friday. Matt Erkkila was hired as a web programmer, and quickly progressed to senior programmer and then Web Systems Manager with a couple of programmers under him. He single-handedly coded our entire CDA (Content Display Application) and was responsible for keeping all of the systems used by the Internet department running smoothly – while he and his team continued to develop new online products.

Matt often liked to stretch his legs after long bouts of programming or working on system problems, and this was often construed as ‘Matt was fooling around’. We’re a mid-sized family owned company and the only other one with a Computer Science degree who had done the type of programming he was doing was me. I understand the need to walk away from the computer once in awhile to socialize with your peers and give your brain a chance for that ‘aha!’ moment that you need to finish what you’re working on.

I think this finally started to become apparent to others during the past two weeks when we were debriefing Matt of all the projects and systems he was responsible for. Our online world worked relatively smoothly for the past three years due to the unseen magic that Matt worked on all the systems he touched. I am very pleased with what we were able to accomplish in the past month to meet the overwhelming traffic increase we have seen since we moved to the hosting company. The CDA has proven to be extensible and solid after some tweaking. Most people would have basically given up after they knew they were leaving the job (and the state), but Matt worked tirelessly trying to get the systems in a state he would be proud of.

So a big shout out to Matt – thanks for the hard work, keep in touch, and see you in San Francisco (you owe me dinner when I come out).

If you’re in SF, Matt accepted a job at Digg – give him a ‘sup’ if you see him.

Technorati tags: , , ,

Registration Update

As you may or may not know, we are currently rolling out registration on all of our websites. This will allow us to gather more accurate stats for our advertisers about the type of people who view our sites (and their ads). We currently only have registration on one website – which only has a dozen or so content items restricted. It has been up for about a week and has 18 registrants, some of who are employees.

Registration is not difficult technically to roll out but it does involve quite a bit of communication between each of the publishers, the circulation department, and our registration consulting company (which is the same as our print fulfillment company). The publisher has to choose what content is behind registration, circulation has to make sure the proper demographic questions are in the database for each publication, and the consulting company has to add these questions in and then give us the id numbers that will be returned in their XML.

We also have one paid publication which uses a different print fulfillment company and so we have the added pleasure of not only coding for a paid model but also to incorporate the other fulfillment company. Luckily the other company is very experienced with online registration and also very attentive and responsive. Our current problem is making sure that our first registration company gets everything together to incorporate this other process. Oh, and we have a deadline of August 11th – which includes launching this last site on Vignette v7.

Things are looking ok on our side so far. And we can meet the August 11th date even if the registration system isn’t quite hooked up all the way to the second registration company. We either launch the site without registration, or put in registration but no one can access the restricted content. We’re going to shoot for August 7th as our launch date to leave us time to clean up any mess there may be. Wish us luck with another deadline pulled out of the hat.

Too Much Traffic? Update

A couple of weeks ago we migrated our Oracle database from 9i to 10g and moved it to a faster, more powerful server. The migration went extremely well, especially considering these types of moves are usually planned out very carefully, well in advance.

We did end up having a few minor hiccups after the cutover – one related to a runaway query that had been tweaked in 9i but which 10g thought it could execute without the 9i tweak. Adding a query execution plan fixed that problem and we were running along smoothly. It is nice to have a DBA (database administrator) who seems to know what he’s doing – and the DBA-in-a-box tools that come with 10g made things much easier to track down.

We still were having problems with our CDAs (Content Display Application) after the database migration – with hardly any work being done on the database after it got its new home (and we segregated NetTracker to its own database and server). Our lead programmer was able to implement caching of the XSLTs (eXtensible Stylesheet Language Transformations) which take the XML and transform it into HTML which the browser can read. Each webpage has probably close to 100 XSLTs that are run on it to form it into one HTML document. We treat the XSLTs like templates, which are reused throughout our sites and are edited very rarely. Caching them made a lot of sense – and even with a 5 minute cache, we were able to decrease the load on the CDA servers by ten-fold and as well as decreasing our full-page load times from 6-10s down to 3-4s. I need to check on our monitoring to see how we’ve been doing at peak traffic times (typically around noon EDT for us), but this was a major improvement.

Our front end web server (MS IIS) was also replaced with Apache since we were seeing some inconsistencies with mod_jk running on the windoze box and talking to Apple OSX and RedHat in the JBOSS CDA cluster. Our next step will be to look at possibly adding more servers into the CDA cluster with a load balancer and another apache web server on the front end.

Overall a nice team effort.

Technorati tags: , , , , , ,

Wil Wheaton as Kirk vs. Picard

  • Kirk vs. Picard: Wheaton Channels Kirk and Picard In a new video on the KvP site Wil Wheaton acts out moments from the five finalist scenes from the first round of the KvP contest (Guardian Angles, Enemy Past, From Like Tears in Rain, Wowbagger Scene 1 untitled, and the winner: Guardians of Forever).

Wil does a pretty good Kirk and his Wesley bit is hilarious.

Internet Radio Stays?

  • Listening Post: No “Promises” From SoundExchange’s Jon Simson – Jon Simson, executive director of SoundExchange, told members of Congress and representatives of webcasters yesterday that webcasters who were in compliance with reporting rules and were in negotiations with SoundExchange didn’t need to shut down in light of new royalty rates.

While not necessary a stay of execution, this promise holds hope for all Internet radio stations which are in royalty compliance. I’m glad Radio Paradise will continue playing… at least for now.

Technorati tags: , ,

Too Much Traffic?

We have been having a lot of problems with our web systems at work lately. A couple of months ago we moved all of the servers to an outsourced hosting company with a much larger pipe to the Internet. Since that time, our traffic has grown at least 10-fold. While this growth is good in the long run, it is putting strain on systems not meant for that type of traffic. We planned in quite a bit of scalability into our systems, but have not been able to put in many of the upgrades due to personnel and political issues (mainly lack of approved monies). This weekend we will be putting in a new database server, running on linux instead of Windows and upgraded to Oracle 10g.

The migration process itself should be fairly smooth (keep your fingers crossed). Unfortunately we don’t have the time to plan out everything properly, but all the applications which connect to the database have only one connection string to change (or so the vendors tell us). Luckily, we still have the current database as a fall back if something goes awry.

Our next step is to look at caching at the XSLT and object layer on the application servers. Our Web Systems Manager was able to finish his testing and debugging of the XSLT caching, so we just have to deploy it – we’ll be waiting until after the database move to keep things clean.

Also of concern – which OS is going to be the best choice for the CDAs (content display application) or application servers? We were running Windows and that was ok until we migrated the servers. Right now we’re running Apple OSX servers, which seem to be doing fairly well. The problem is that the traffic increase was so drastic that we may have to throw a large chunk of hardware at it – including more application servers, probably increased to at least another complete cluster, which means another apache server out front and a load balancer to send the traffic to the right cluster.

Lots of different problems to attack all at once which is difficult even if having too much traffic is the type of problem most people want to have. My deepest concern is solving the problem(s) well enough that we don’t lose that traffic jump as it turns elsewhere when we can’t serve their requests.

Technorati tags: , , ,

Project Update

Still working on my side project – been doing a lot of writing, thinking and organizing. I have promises that I’ll be getting some of the design work I’m waiting for this weekend. I’m planning on doing some more work on setting things up this weekend.