Technology Aligned Teams vs. Platform Aligned Teams

As a consultant, I had the opportunity to work within a wide variety of organizations. From start-ups to Fortune 100; from banking to transportation. It was interesting to see the wide variety of organizational alignments devised by different companies in different industries. Despite the endless variety and nuance, there seemed to be a couple of common themes across companies despite the specific industry in which they operated.

Technology Aligned Teams

Large IT organizations are often divided into teams based on the technology they are responsible for supporting. A networking team manages switches and routers, a desktop team takes care of workstations, a storage team manages large disk arrays, SANs, and NAS, and so on.

Such an alignment used to make sense. The technologies being deployed were often new and poorly documented, products were complicated to setup and required a significant investment in technical knowledge, and diagnosing issues was almost a black art for some areas. This situation suggests the need to hire engineers who are narrowly focused on very specific technologies. And that's what many companies did. (And still do.)

The benefits and drawbacks to this approach are well known. Benefits are supposed to include cross-training, a deep bench, best-of-breed practices, and an enterprise-level perspective on architecture, procurement, and issue resolution. Drawbacks include IT departments requiring architecture review boards, one-size-fits-all configurations and change management windows that may not suit the needs of individual business unit. Implementations for fast moving products end up waiting in line behind slower business units, and unintentional dependencies emerge across logically separate product lines. For example; load balancers that are shared across multiple product lines. One product needs a load balancer change implemented, but must wait for the weekly change window; otherwise, other products could be negatively impacted by the 'unscheduled' change.

In reality, the cross-training seldom happens. Each team has one or two go-to people surrounded by a bunch of extra hands. The idea of having multiple network engineers is great, but when things are in the ditch, most companies grab the go-to person to save their hide. Over time, teams become increasingly focused on the technology they manage instead of the business products the technology is supposed to be enabling.

Platform Aligned Teams

An alternative approach in some organizations is to create mini-IT departments assigned to a specific product or business unit. The team might include a server and database administrator, a networking engineer, a web server expert, and other skills needed to support a particular platform or business unit. Many times, the people have multi-faceted skills like the combination UNIX/firewall/load-balancer/apache/networking administrator or the DBA/performance tuner/SAN/backup engineer.

Again, the benefits and drawbacks to this approach are also well known. Because this mini-IT group is often allowed to operate with more latitude compared to a centralized IT department, they are better positioned to respond to the needs of the business (read: customer).

While this approach has many benefits at a product or business unit level, analyzing the enterprise wide cost of such an approach can paint a costly picture. Different groups may select different vendors, economies of scale such as price negotiation are diminished, and not all mini-IT groups have the same level of skill.* Risk mitigation may require more focus to avoid the scenario where configurations introduced by one group have a very detrimental effect across an entire company. For example; not properly maintaining a strong security posture or disrupting a network routing algorithm.

* In the past, it was very difficult to find that magic collection of engineers that the wide variety of experience necessary to build a strong mini-IT group. The maturation of most products and the increasing number of renaissance engineers (engineers with skills across a wide variety of disciplines) has definitely improved the viability of this approach.

Summary

Between the two approaches described so far, my experience would suggest platform aligned teams are more practical and deliver better results for customers. I think this is especially true in a turnaround scenario (rescuing a failing division or subsidiary) as well as business units focused on growth or new markets. Products and business teams that exist in a saturated or slowly shrinking market might be okay with the traditional technology aligned approach -assuming the goal is to simply ride out a cash cow or avoid pursuit of significant strategic improvements in order to prepare for a sell-off.

Next time: What if there was an approach that combined the best of both worlds?

Implementation is More Important Than Product Selection (Part 2 of 2)

In my previous post, I described a common scenario an organization experiences when acquiring a new IT component. It's easy to point out flaws and criticize companies when you're a single actor in the equation, but the fact is, no single person possess all information related to all of the competing business drivers. I've heard plenty of self-righteous water cooler conversations from smart people who describe how things should have been done. Their theories are good and certainly suggest a better outcome than the most recent experience, but they usually lack pragmatism.

Over the course of my career, I have fulfilled all of the roles described in the earlier procurement scenario. I was the IT engineer asked to sit with the vendor to watch everything they did and learn over their shoulder -only to have my manager redeploy me to a new fire. I've had to be the manager responsible for pulling someone off such an engagement because they need to address an urgent issue elsewhere within the company. I've been the professional services expert that's hired to come in, setup the component, configure it for operation, and cross-train the existing staff. I've taken people on the golf trips and been taken on the golf trips. I'm also a purchasing decision maker responsible for the budget and the director that's pounding on the table because “we need it in production now!”

Participating in this scenario from so many different angles has taught me a lot. A couple of things that come to mind here include...

Make key decisions faster

Most IT components have matured to a point that they are capable of supporting all of the use case scenarios likely to be encountered by a typical company. Obviously, there are exceptions, but whether it's a firewall, switch, server, storage array, or other device; most companies will do fine selecting from any of the top three vendors in any category.

Not even a perfect product selection will succeed if the underlying architecture is flawed or your people don't know how to manage it.

The point is; organizations can invest too much time equivocating on which vendor to select. There's a point after which additional research, discussions, tests, and other activities simply do not improve the quality of a decision.

Allocate time where it will have the biggest impact.

In the previous example, a new component was evaluated, selected, and installed over the course of six months. The team invested five months in product selection and a week or two in implementation. The motivation is obvious; “If we're going to spend blah dollars on this thing, we want to get it right.” Again, most organizations invest too much time trying to select the perfect product for their needs when any of the top three vendors will do fine.

Instead of investing 90% of the time on selection and acquisition only to leave 10% for implementation, suggest spending at least as much time on implementation as you did on selection. In the previous example, suggest investing two months on product selection, acquire it, and move forward. Then spend two to three months on configuration and testing. Let all of the engineers spend time with the product and truly learn it. Let them learn how to build it up from scratch by themselves. Force it to break. Understand what can make it break and how to repair it. If it's hardware, actually go through the hot-swap procedures of all sub systems. Make sure your team knows this thing inside and out since they'll be supporting it once it goes into production.

Support your team, and your organization, by allocating and protecting the time necessary to truly learn, correctly configure, and confidently deploy new technologies. These aspects of deployment are far more critical than product selection and will directly impact an organization's ability to effectively leverage the technology for years to come.

Implementation is More Important Than Product Selection (Part 1 of 2)

Far too often, I see IT professionals invest a significant amount of time evaluating products for a specific solution. Vendors are drilled on the capabilities of their products. Engineers fire off questions they hope will make them look smart in front of their bosses and point out holes in the vendors' solution; “So, you're saying your firewall appliance doesn't support an enterprise wide CORBA bus!” Never mind the engineer knows perfectly well they're company doesn't have a CORBA bus, doesn't plan to use such an architecture, and nobody else does either. Demo units are brought in, tests are haphazardly conducted, the field is narrowed to top industry leaders, fiscal year-end concessions are made by a vendor, someone goes on a golf trip, and finally; a product is selected.

After five months of meetings and evaluations, the money has been spent. Management wants the shiny new product in use immediately. “We didn't spend blah dollars on this solution to have it sit around on a shelf. We need it in production now!” Because the local folks aren't experts yet, the vendor supplies a professional services contractor (probably a sub contractor from one of their resellers) to “install, configure, and perform knowledge transfer” with the team.

The PS guy shows up and is guided to the conference room that's been reserved for the whole week so the team can watch and learn how to perform installation, configure the device, and receive knowledge transfer. Four hours into day one, the local folks are pulled off to fight some IT fire. The PS guy forges ahead, begs for business requirements, completes the work as best he can, writes up some documentation, and flies out early on Friday.

The implementation is finished.

At this point, the company relies heavily on a piece of equipment their own people don't understand very well. The device is in the critical path of business operations; therefore, it's “hands off!” Changes are approached timidly with fingers crossed instead of confidence. Over time, the poor understanding leads to unintentionally poor configurations. The device begins to act unexpectedly. It begins to develop a negative reputation because it's “unstable” or “unpredictable” or “doesn't support feature x”.

Of course, nobody suggests it's being managed poorly -that the people don't know what they're doing. That might hurt somebody's feelings. Besides, they know they have good people. The folks have a good attitude, work long hours, and put out a lot of fires. “That piece of junk is causing us all kinds of headaches. It needs to be replaced!”

And the whole cycle starts over again...

Next Post: Is there a better way?

DevOpsATL Meetup Group Launched!

It's been great working within the Atlanta IT community. I've had the pleasure of working with some very talented and professional colleagues. Recently, I had the opportunity to reconnect with an old friend. This chance meeting eventually lead to the creation of the DevOpsATL meet-up group...

Earlier this year, I signed up to attend the 2010 O'Reilly Velocity Web Performance and Operations Conference. As the event drew closer, I checked the conference sign-up list to see if anyone I happened to know would be going. To my surprise, there was an old name I hadn't seen in years. While catching up at the conference, he introduced another friend of his and the three of us started discussing the idea of creating a meet-up group to serve the Atlanta community. A few weeks after the conference, everyone actually followed through and created the DevOpsATL group on meetup.com.

All of the DevOpsATL organizers are great to work with and have a lot of practical experience to offer. We've had a couple of meet-ups already and I am quite impressed at the quality of people who've decided to join the conversation. We're really looking forward to giving back to the Atlanta IT community, so if you're in the area during one of our meet-up dates, stop in for some compelling conversations.

(too) strong passwords = less security

There are many effective ways to secure access to computer systems, but I suspect most users' access to most computer systems is still largely controlled by a password. With all of the great alternatives to single passwords out there, I wouldn't defend this fact as acceptable; rather, it's simply very common. And of course, if a user is required to utilize a password, it may be written down somewhere. Again, not acceptable, but quite common.

Not me! I'm smart. I work in IT. I've secured networks and servers myself. I would never write down my password!

So, information security professionals insist passwords should meet certain strength requirements so users don't use "password" for their password. Okay, strength requirements seem like a rational requirement: minimum number of 8 characters, includes a mix of alphas and numbers, and must be changed every 90 days. Their boss says; "Good work. It meets our audit requirements. Keep it up."

As an IT professional who generally supports efforts to secure our assets, I go along. I dutifully dream up a password and remember it without writing it down. Then, I log into the 26 different systems that authenticate me separately and set my new password. (Yes, it would be more secure to use 26 separate passwords, but I'll save my rants on the history (pipe dream) of centralized authentication and single sign on for another day))

So, those information security people figure they should make the password requirements even stronger: minimum number of 8 characters, includes a mix of at least 2 lowercase alphas, 2 uppercase alphas, 2 numbers, and 2 special characters, and must be changed every 60 days. Their boss says; "You guys are great! I can tell the auditors we're even strongerly protected!"

Groan. Okay. I dream up an algorithm to use that fits the criteria and begin setting my 26 accounts. But, three of the systems don't support special characters. Doh! Now I'm forced to do password forking. I adjust my algorithm to account for the lack of special char support and know what to substitute when I hit those systems.

So, those information security jerks figure they should make it even stronger so they can justify their job; minimum number of 10 characters, includes a mix of at least 2 lowercase alphas, 2 uppercase alphas, 2 numbers, and 2 special characters, must be changed every 30 days, and no character can be re-used in the same position for 12 months (ie: AppL#14! cannot replace 0p&n23T# since a "p" is re-used in the second position). Their boss says; "You guys are totally rad! I bet I can score a date with that cute auditor with security hotness like this!"

Groan. My algorithms are shot. I find four systems that can't support passwords over 8 chars. Of course they only partially overlap with those that don't support special chars. Now I need 4 separate passwords every 30 days.

Then, those information security jerks disallow any repeating chars (like the "pp" in AppL#14!). The interval is reduced to 28 days, so my first of the month calendar reminder no longer works. The password event starts arriving earlier each month. Their boss says; "We love you! Which certification would you like the company to pay for you to pursue?"

I find systems that allow over 8 chars, but only enforce rules on the first 8 so a special char in position 9 isn't recognized. I invest hours writing expect scripts to automate this monthly hell, but some systems are difficult to script against java GUIs -not impossible, just not worth the time.

So, they do some more stupid stuff. Their boss says; "You're promoted!"

I give up. The Post-It note goes in my wallet.

I suspect there are others that share my frustration. I'm willing to support security efforts, but at some point, they become completely impractical and, ultimately, weaken the overall security posture of the assets they are trying to protect.

Have you experienced anything similar?

Setting Up a Virtual Frame Buffer to Support Oracle Reports Server

Article originally published by John Christian as a Sun BigAdmin Tech Tip)

Introduction

As the name implies, Oracle Reports Server is used to generate pretty reports. In some environments, these reports are delivered on an as-needed basis using X-connections to X-Servers. In other environments, these reports are run on automated batch schedules.

The funny thing is, Oracle Reports Server insists on connecting to an X-Server to function, even if it's running in batch mode. A common workaround is for the DBA to start an X-Term session on the server console and have Oracle Reports Server connect to it. Problems predictably arise when a sys admin comes along and "cleans up" the CDE, logs out, and so on, and the X-Term session is killed. Besides, what do you do if the server is headless and not running a local X-display at all? Route the display to another server or workstation that is running an X-Server? Obviously, these workarounds lead quickly down the path of "kludgedom".

Xvfb (in combination with twm) provides a more elegant solution of a more civilized age. The primary use of the X virtual frame buffer was intended to be server display testing. The X community has found many novel uses for Xvfb, including testing software clients against unusual color depths and screen configurations, doing batch processing with Xvfb as a background rendering engine, load testing, as an aid to porting the X server to a new platform, and providing an unobtrusive way to run applications that don't really need an X server but insist on having one anyway ... like an Oracle Reports Server!

To provide Oracle Reports Server with the X-display it desires, you can configure the Oracle Reports Server startup script to launch the Xvfb daemon to provide a virtual display device (frame buffer), and twm to provide a window-management framework.

Demonstration

Xvfb and twm are bundled with the default X installation on the Solaris Operating System. Below are the steps to try out Xvfb manually for functional verification. Some of the examples assume you have a CDE running on the host. While using Xvfb is not necessary, having an existing X-Session running makes the examples easier to follow.

First verify that Xvfb is available on your system. The default installation directory is /usr/openwin/bin.

$ ls /usr/openwin/bin | grep Xvfb
-rwxr-xr-x 1 root bin 290 Jul 10 2003 Xvfb

Let's have some fun with Xvfb.

$
$ PATH=$PATH:/usr/openwin/bin;export PATH
$
$ DISPLAY=:0.0;export DISPLAY
$ xclock

At this point, the xclock pops up on the local host console screen. (Press Ctrl-C to kill xclock.)

Now start the X virtual frame buffer:

$ Xvfb :5 -screen 0 1600x1200x32 &
6897
$ Warning: There is no XDISPLAY information for display 5.
Server is using XDISPLAY information for display 0.

Don't worry about the warning now; you eliminate it in the final configuration.

Reset the DISPLAY environment variable to point to the virtual frame buffer:

$ DISPLAY=:5.0;export DISPLAY
$ xclock

xclock runs with no errors. Although you can't see the clock, the lack of errors indicates that the clock has connected to the virtual frame buffer and is working.

What About twm?

The twm process provides a window-management framework for X applications that require it. Most applications -- xclock for example -- do not expect a windows-management framework to be present and function fine without it. Some applications -- Oracle Reports Server, for example -- expect a window manager to be present and do not function properly without it. The twm process is a part of the final solution.

Why Use :5?

Why would you use :5 for the virtual display setting? If a host has a video card and monitor installed running CDE (or another desktop GUI), chances are, display :0 is already in use. If a secondary video card (with monitor) is attached (very rare these days), it will use the :1 display. The :5 setting was arbitrarily chosen as a value that was high enough to be quite unlikely to conflict with existing display variables.

Final Configuration Using Xvfb and twm: OWconfig File Changes

Remember those warning messages you saw earlier when testing Xvfb? While not fatal, they should be cleaned up. To set up the :5 virtual display properly, edit an additional entry into the /usr/openwin/server/etc/OWconfig file, as shown in bold below.

# X Display
class="XDISPLAY" name="0"
coreKeyboard="IKBD" corePointer="IMOUSE"
dev0="/dev/fb";


# Virtual X Display for Oracle Reports Server
class="XDISPLAY" name="5"
coreKeyboard="NKBD" corePointer="NMOUSE"
dev1="vfb";


# CG6 display adapter
class="XSCREEN" name="SUNWcg6"
ddxHandler="ddxSUNWcg6.so.1" ddxInitFunc="sunCG6Init";


Oracle Reports Server Startup/Shutdown Script Excerpts

In order to ensure the virtual frame buffer is available for user after a reboot, include lines similar to the following example in the "start" portion of your Oracle Reports Server run control script.

#
# Provide a virtual frame buffer for the Reports Server to connect to
#
# Using display number 5 should safely avoid the common displays 0 and 1
echo "Starting Xvfb..."
/usr/openwin/bin/Xvfb :5 -screen 0 1600x1200x32 &
echo "Starting twm..."
/usr/openwin/bin/twm -display :5 -v &
echo "Setting DISPLAY to :5.0"
DISPLAY=:5.0; export DISPLAY

To ensure that the virtual frame buffer is not running unnecessarily if Oracle Reports Server is stopped, include lines similar to the following example in the "stop" portion of your Oracle Reports Server run control script.


# Clean up Xvfb (virtual frame buffer)
#
echo "Stopping Xvfb..."
/usr/bin/kill `ps -ef | grep Xsun | grep :5 | awk '{print $2}'` > /dev/null 2>&1



Process Verification

The Xvfb command actually results in a process running under a slightly different name. After Xvfb and twm are launched, you should see the following entries in the process table (ps details snipped):

/usr/openwin/bin/Xsun :5 +nkeyboard +nmouse -dev vfb -screen 0 1600x1200x32

/usr/openwin/bin/twm -display :5 -v

Enjoy!