I was hired in early 2000 by 'Inktomi' to work as a Software
Engineer. Frankly, if the truth be known, one of the reasons
I was hired was that the job market was extremely tight at the
time. My training had consisted of a year of fairly intensive
self-study on UNIX systems (primarily FreeBSD), and a handful
of entry level courses. I had previously managed technical
support for an internet service provider in Portland, OR.
Inktomi's Release Engineering team needed someone to focus on
assisting one of their divisions with multi-platform build
requirements, and perform system administration on various
Unix systems including Solaris, Irix, OSF/1 (Tru64), FreeBSD.
My first task was to integrate Linux and HP-UX.
'Inktomi' was, at the time of my hire, the predominant search
engine company (pre-google), and they had a very hot newish
offering called 'Traffic Server' which was a caching proxy
server for internet traffic. At that time, there was no where
nearly as much fiber in the ground and there was a great need
for such a solution. Additionally, every company in the sector
was swamped with venture capital so we had many customers who
purchased the software and didn't even get around to installing
it. Many did, however, and so we had plenty of legacy
support considerations. In short, the growth rate of
the company was phenomenal with personnel being added to the
existing divisions, and other companies being purchased and
integrated.
After a little over a year, it was clear that major changes
were in front of us. As the first round of lay-offs hit, I
was retained, given a pay increase on the order of 35%, and
promoted from a rating of 'software-I' to 'software-III'. On
my performance review, I received a 'far exceeds'
which was the highest rating. I later learned that this was
quite rare though I didn't realize it at the time. Since
'Inktomi' is not completely defunct in all senses of the word,
I will share this performance review with the reader only at
special request. I am quite proud of it and believe that I
earned it. I took pictures of the documents with my camera and
converted them to two-color .png files so the size is reasonable
while they remained vaguely legible.
(Performance Review provided at special request)
There are a number of my projects and achievments mentioned
in the review. I won't repeat them, but I must add that they
were all very minor compared to what I later achieved. For one
thing, each new development automated a portion of my work-load
and freed me to focus on more important things. It is critical
to understand that most of these efforts tied into one another,
often to a very high degree. Here is a partial list of those
which I developed _after_ the performance review which
coincided with the first round of layoffs and my becoming the
principle architect in my group.
After that, I was several more times retained as a 'key
employee' until the whole division was down to 8 people (from
probably around 800 at a high point!) One of the task I
undertook was to build a very much reduced system which
would allow one large customer to continue development of a
product they had purchased from us. Aside from that there
was not a whole lot to do and I mainly focused on collecting
hardware in case Yahoo! (who had purchased Inktomi for their
web-search technology...a different division) decided to continue
development of any of my division's products.
After I left, Yahoo! did indeed decide to continue development
of one server product and hired a small team to do the
development work. I came back on contract and built a tight
development framework which included the following features:
Near the end of 2003, I got a contract to consult on a
configure/build system, and that lead to an extension of the
contract to work on implement some of the ideas we discussed.
I do not feel at liberty to name the company. They were small
and young, but old enough to have been strained by issues with
their existing system. My total time on this contract was 6
weeks.
The system that emerged had some very interesting features
that I am pleased with. Their product consisted of a set
of quasi-independent projects, some customized open-source
code, and some non-customized open-source code. What I ended
up with was a means of arbitrarily and easily defining any
set of code bases where each was from an arbitrary branch.
Mutual dependencies were automatically detected and merged, and
instances where multiple versions of the same code base were
defined were readily recognizable and resolved. Furthermore,
the build procedure from the developer's point of view was very
straightforward, uniform, and comfortable, and he/she could
enter the hierarchy at any level (if, say, they had checked out
the definition for large project.) I used GNU Make as a driver
to handle the dependencies and it was fast and accurate.
Do to my previous experience, I specifically designed in the
flexibility to drive a multitude of different build types, and
possibly pre-built components in instances where that might
make sense. Also, I was careful to ensure that multi-platform
simultaneous builds would be possible.
My main regret about this project was that although I
accomplished more than I initially thought possible and
demonstrated real-world functionality, there was simply not
enough time to develop a number of features that I envisioned as
being possible and designed in support for. I also did not get
every module ported. For multi-platform efficiency reasons (and
development convenience), I'm a stickler for separate-source
builds and much open-source software comes with defective
Makefiles which require work.
Tools:
My focus has been entirely on Unix, and that is where
my interests lay. I never had a Windows workstation or a
particularly active part in Windows system design so my skills
are adjusted accordingly toward Unix and away from Windows
environments.
I am proficient with Bourne Shell (driving sed,
awk, etc.) Because of my work with autoconf (which
uses m4 macros to generate portable Borne shell script,) and a
general need to have cross-platform compatibility, I ended up
using Bourne shell a lot.
For more complex tasks, when the situation allows, I tend to use
Python. I appreciate it's object oriented support and
focus on maintainability and have found both of these to pay
great dividends.
Occasionally I found it necessary to use C to write
certain utilities, but do not consider myself a C programmer.
Similarly, I occasionally needed to do some C++ porting, but
usually consulted someone more competent when possible. I spent
a great deal of time _building_ programs in said languages and
working through compiler issues, however, and am proficient at
solving associated problems.
My experience with Perl and tclis limited to minor bug fixes of, and additions to existing tools that I use. I used Python for original work requiring a language in that category.
I am proficient with CVS administration and have devised various solutions making use of an understanding of it's internal implementation. Unfortunately, I have no experience with any other revision control system (RCS excluded.) At my next opportunity, I do intend to explore 'Subversions' which appears to be a promising open-source CVS replacement.
Other tools I made significant use of include autoconf,
autoconf, ssh, rsync, procmail,
tinderbox-II (a mozilla-developed display system),
ViewCVS, curl, make tar, and
numerous similar basic Unix tools.
System Administration:
I have always set up my own Unix workstation and it is not
practical (or generally possible) to burden 'helpdesk' with the
task, so I can solve basic sys-admin issues like Unix X11 setup,
print-serves setup, etc. Additionally, I use FreeBSD which is
not as polished as most of the Linux distributions and generally
requires more hand work and understanding.
I am especially familiar with automounters (both 'autofs'
and 'amd') and filesystem design in multi-platform environments.
Also with NIS and NFS to the extent that they are somewhat
inter-related, at least in an environment which is vaguely
sophisticated.
I've worked out the details of using 'mail' for certain things
(like launching certain processes) and using something else in
it's place when it is not appropriate (like feeding huge amounts
of data to Tinderbox-II using 'curl' instead of e-mail, and
using other methods of feeding CVS monitoring databases.) In
general, I pride myself in finding novel solutions to problems,
and often before they become problems.
General:
I consider myself to be a systems architect first and foremost.
To my non-computer literate friends, I describe my job as being
analogous to a person who develops and sets up the machines in
a manufacturing facility. I enjoy such work and take pride
developing systems which are efficient, reliable, maintainable,
and coherent.
I am very interested in automating processes to the highest
level practical. Ideally, once a system is in place I would
like it to run without intervention. For instance, the build
system I developed took care of various housecleaning tasks
automatically that used to consume a significant amount of time.
The less a system needs to be touched, the easier it is to
forget the details of how it works, thus, I believe in complete
and legible feedback (for debugging) and good documentation.
I'm not adverse to taking on tangential responsibilities when
that is what it takes to get the job done. Arguably, our
core system administration group at Inktomi should have been
responsible for some of the sys-admin details that I performed,
but in reality I was the (only) one in a position to appreciate
the importance of certain facets of the job, and it just
worked better if I handled it. On the other hand, I did try to
push back when the shoe was on the other foot. As an example,
if a developer wanted me to handle a third-party element and
it would require me to learn significant aspects of how it
interacted with our code, I would argue against it. Though I
would have liked to learn these types of things, it would have
been inefficient and interfered with other important projects
which I was chronically anxious to attack.
I have spent considerable time contemplating the role of Release
Engineering, and the things I've observed about the Software
Engineering environments I've been associated with. At Inktomi,
I sat on a number of 'core teams' where I absorbed a great deal
of information about both the product and the process (at the
expense of loosing time dedicated toward my 'real work'.) In
retrospect, I believe that it was of critical value. Here
are what I consider the most interesting thoughts on these
matters:
Firstly, I see Release Engineering as being _ideally_
situated to mediate between the often conflicting demands of
'Development', 'QA', 'Project Management' and 'Support'. A good
Release Engineer should be always mindful of the stresses on
each of these parties, and always on the lookout for ways to
reduce the problems that each may face.
Secondly, and more of a 'system analysis' issue, I think that it is imperative for efficiency and quality to pay close attention to how tasks and responsibilities are assigned. I've touched on this above. That is, as a Release Engineer, tended toward being more involved in system administration, and less involved in product development details. In this case, understanding the system administration details that were important to me would require an sys-admin to know a great deal about my requirements. (i.e., it may _not_ be desirable to patch a certain machine's OS for reasons related to reproducibility.) It would be inefficient, if even possible, for a sys-admin to understand the specifications and the reasons behind them. It was much more practical for me to learn the the basic sys-admin and system design skills that I needed to perform these tasks. On the other hand, while building third-party libraries is something that a Release Engineer should be quite familiar with, integration into the product often requires a good bit of understanding known most completely to the developer who determined that the inclusion was a solution to the problem he/she was facing. In this case, I would tend to work with the developer on integration issues, but leave it to the developer to work through build details which related to product/library interoperability. This is one simple example of an issue that crops up regularly, and addressing logically pays dividends in many areas throughout time. Release Engineering is well positioned to recognize a number of these.
- more to follow -