Automation for the people: Deployment-automation patterns, Part 1

Patterns for one-click deployme

Software deployment is often treated as a necessary evil that can becobbled together before going live. But you can and should applysoftware-engineering principles to deployment as well as to other partsof the development cycle. When manually applied, deployment is arepetitive and error-prone process. Just as you can automate builds toreduce errors and speed software development, you can automatedeployment processes to reduce errors and expedite software delivery.

In an earlier Automation for the people installment, "Speed deployment with automation,"you learned a technique for remotely deploying software into multipletarget environments. This two-part article takes theautomated-deployment discussion to a higher level. Just as there arepatterns for software development, there are patterns for deployment,and I've been cataloging them over the past few years. Here in Part 1,I'll introduce eight deployment patterns and provide examples for them:

• Managing all configuration files in a centralized Repository, which makes it possible to generate working software using a Scripted Deployment
  
• Scripted Deployment, which scripts all deployment actions so there's no need for human intervention when a deployment is executed
  
• Single Command, which reduces deployment complexities and ensures a Headless Execution of the deployment process
  
• Tokenize Configuration, which provides a repeatable way of injecting variable information into configuration files
  
• Externalized Configuration, which makes it simple to enter information only once that varies between target environments
  
• Template Verifier, which helps to ensure that all target environment properties are the same
  
• Headless Execution, which provides a secure way of accessing multiple machines in an automated process
  
• Unified Deployment, which promotes a single deployment script that can be run in many target environments

You'll learn about even more deployment patterns in Part 2.

About this series

As developers, we work to automate processes for end-users; yet,many of us overlook opportunities to automate our own developmentprocesses.To that end, Automation for the people is a series of articles dedicated to exploringthe practical uses of automating software development processes and teaching you when and how to apply automation successfully.

Figure 1 illustrates the relationships among the deployment patterns covered in this article:

Figure 1. Deployment-automation patterns

I'll discuss each pattern in turn. As you read about them, you'llgain an understanding of the interrelationships illustrated in Figure 1.

Committing all files to a version-control repository

Name: Repository

Pattern: All files are committed to version-control repository — in the deployment context, all of the configuration files and tools.

Antipattern: Some teams put this information on a shareddrive with controlled access. Others may keep the information only ontheir machines and then copy it to target environments.

As a general rule, I recommend that development teams check in allof their files necessary to create working software. Sometimes thereare exceptions to this rule, but not often. In the context ofdeployment, some teams erroneously look at servers and serverconfiguration as fixed assets — that they do not change. Although, attimes, there are constraints on checking in large binaries, theconfiguration, database scripts, and all build and deployment scriptsshould be committed to the version-control repository. Using theRepository pattern greatly assists in the patterns I'll discuss next:Scripted Deployment and, particularly, Single Command.

---

Back to top

Scripting all deployment processes

Name: Scripted Deployment

Pattern: All deployment processes are written in a script.

Antipattern: Some may manually configure tasks such asinstalling and configuring a Web container. Others may use theGUI-based administration tool provided by the container to modify thecontainer based on a specific environment. Although not likely, itcould be simple to modify configuration manually the first time, but itdoesn't scale when you perform multiple deployments to many targetenvironments several times a week or more. Moreover, a GUI-basedadministration tool can be very helpful in deployment — the first timeyou use it. However, this approach also does not scale and iserror-prone, because many people must perform these procedures againand again.

Listing 1 demonstrates the Scripted Deployment pattern by automatingthe process of (re)starting a Tomcat Web container through a scripteddeployment. This process is written using the Apache Antbuild-scripting language.

Listing 1. Example of starting a Tomcat Web container

<available file="@{tomcat.home}/server/@{tomcat.server.name}/bin" 
   property="tomcat.bin.exists"/>
<if>
  <isset property="tomcat.bin.exists"/>
<then>
  <echo message="Starting tomcat instance at @{tomcat.home} with start_tomcat" />
  <exec executable="@{tomcat.home}/server/@{tomcat.server.name}/bin/start_tomcat" 
   osfamily="unix" />
</then>
<else>
  <echo message="Starting tomcat instance at @{tomcat.home} with startup.sh" />
  <exec osfamily="unix" executable="chmod" spawn="true">
    <arg value="+x" />
    <arg file="@{tomcat.home}/bin/startup.sh" />
    <arg file="@{tomcat.home}/bin/shutdown.sh" />
  </exec>

  <exec executable="sh" osfamily="unix" dir="@{tomcat.home}/bin" spawn="true">
    <env key="NOPAUSE" value="true" />
    <arg line="startup.sh" />
  </exec>

    <exec osfamily="windows" executable="cmd" dir="@{tomcat.home}/bin" spawn="true" >
      <env key="NOPAUSE" value="true" />
        <arg line="/c startup.sh" />
    </exec>
    <sleep seconds="15" />
    </else>
  </if>

 

By scripting this process, you eliminate the need to click throughthe GUI administration console that Tomcat, in this case, provides.What's more, because it's scripted, it can be run by a headless processas part of a comprehensive automated deployment.

---

Back to top

Getting deployment to run from a single command

Name: Single Command

Pattern: Deployers, or headless processes, can type a single command to generate working software for users.

Antipattern: Some deployment processes require people toenter multiple commands and procedures such as copying files, modifyingconfiguration files, restarting a server, setting passwords, and otherrepetitive, error-prone actions. If they are lucky, they have access tostep-by-step documented procedures for performing these actions.Regardless, requiring humans to perform deployment proceduresintroduces the risk of errors and causes time bottlenecks that delaythe release of software into the multiple target environments.

Development does not equal production

The fact that software works in the development or QA environmentdoes not mean it will work in production, because of the potentialdifferences between the environments. This is why it's important toscript all aspects of deployment.

When you write a deployment, your clients are often others on yourteam, organization, company — or even a machine. The more complex it isto run the deployment, the less likely someone or a headless processcan successfully execute it without introducing errors. A simplesingle-command example for running a deployment is shown in Listing 2:

Listing 2. Single-command deployment using Ant

ant -Dproperties.file=$USERHOME/projects/petstore/properties/dev-install.properties /
  deploy:remote:install

 

The command in Listing 2 executes an Ant task called deploy:remote:install,passing along an environment-specific .properties file, to deploy thesoftware remotely on other machines. This task performs actions such assecurely copying files using the Secure Copy protocol (SCP); securelyexecuting commands on remote machines via Secure Shell (SSH);installing, configuring, and restarting Web containers; and a host ofother processes — without human intervention. Obviously, a user canenter the command, but because it's so terse, it can be easily run by aheadless process, such as a Continuous Integration or build-managementserver.

---

Back to top

Injecting variable information into configuration files

Name: Tokenize Configuration

Pattern: Token values are entered into configuration filesand then replaced during the Scripted Deployment based on ExternalizedConfiguration properties checked into Repository.

Antipattern: Target-specific data is entered into configuration files in each environment.

Listing 3 is an XML file to manage the configuration between a Webcontainer and a database server. In this file, I've placed tokens usingthe @ signs. During the automated deployment process, ascript will replace these tokens with the actual values from theExternalized Configuration files.

Listing 3. Tokenized Web-container configuration file

<datasources>
  <local-tx-datasource>
    <jndi-name>@application.context.name@</jndi-name>
    <use-java-context>false</use-java-context>
    <connection-url>@database.url@</connection-url>
    <user-name>@database.user@</user-name>
    <password>@database.password@</password>
    <driver-class>@database.driver@</driver-class>
  </local-tx-datasource>
</datasources>

 

Tokenizing what will become environment-specific values enables theScripted Deployment to support multiple environments with a UnifiedDeployment.

---

Back to top

Extracting all environment-specific properties

Name: Externalize Configuration

Pattern: All variable values are externalized from the application configuration into build-time properties.

Antipattern: Some hardcode these values, manually, for each of the target environments, or they might use GUI tools to do the same.

Listing 4 shows some of the properties that are often contained inapplication-specific configuration files through the code base. Bycentralizing all variable values into a single .propertiesfile, you can decouple data (the variable properties) from behavior(the deployment scripts). In other words, regardless of the targetenvironment, the automated deployment process runs the same way everytime.

Listing 4. Example properties that are external to application-specific files

authentication.type=db
application.url=http://${tomcat.server.hostname}:${tomcat.server.port}/brewery-webapp
database.type=mysql
database.server=localhost
database.port=3306
database.name=mydb
database.user=myuser!
database.password=mypa$$!
database.url=jdbc:mysql://${database.server}:${database.port}/${database.name}
tomcat.server.hostname=localhost
tomcat.server.name=default
tomcat.web.password=pa$$123!
tomcat.cobraorb.port=12748

 

Often, the values in Listing 4 are scattered throughout source code,server configuration, XML, .properties, and other files. Moreover, I'vefound this data to be duplicated throughout a system, leading tocomplex deployment problems that are difficult to debug. By extractingthis information from this many sources into a single .properties file,you'll reduce the myriad deployment problems that could occur.

---

Back to top

Save yourself with headless execution

Name: Headless Execution

Pattern: Securely interface with multiple machines without typing a command.

Antipattern: People manually access machines by logging intoeach of the machines as different users; then they copy files,configure values, and so on.

Despite its barbaric-sounding name, headless execution is quite aneffective solution when you need to access other machines remotelythrough an automated process. By using a public key infrastructure(PKI), you can orchestrate commands that normally rely on a developer,build engineer, software configuration management (SCM), or operationsperson into an automated solution. In Figure 2, a private-key file isinstalled on the build machine and SSH. Specific values are defined ineach of the target-specific .properties files. This typically includesthe private-key file name and location, the SSH port number, andhostname. The target machines host the public SSH key file to completethe SSH handshake.

Figure 2. Using SSH keys to implement the Headless Execution pattern

If you use this approach, a Scripted Deployment can executeprocesses from the build environment to the target environment (orenvironments) without human intervention.

---

Back to top

Verifying properties are the same across environments

Name: Template Verifier

Pattern: Create a single template file that all target environment properties are based on.

Antipattern: Some use manual verification, trial and error(when deployment fails, check to see why), or keeping files "hidden" ona machine.

The problem is that you need to ensure that you've got the exactsame attributes in every target environment. But how do you verify thisin an automated environment? By using a single template .propertiesfile that all of the target environment files verify against, you canensure that all attributes are the same regardless of the environmentyou are running in. In Figure 3, the build script runs an Ant task thatcompares the attributes (not the values) between thetemplate.properties and the target-specific .properties file(dev.properties, qa.properties, and so on). If it finds a difference,the deployment fails.

Figure 3. Implementation of Template Verifier pattern

Listing 5 shows an example template.properties file like the one illustrated in Figure 3. Notice that it includes only attributes, because the values are irrelevant.

Listing 5. A template file containing attributes, but no values

db.database=
db.username=
db.password=
db.hostname=
db.driver=
db.port=
db.url=

 

Listing 6 shows a snippet from a dev.properties (or qa.properties andso on) file as illustrated in Figure 3. Notice that it includesattributes and values. The values are specific to the target environment.

Listing 6. A target environment properties file based on template file

db.database=brewery
db.username=root
db.password=p@ssword
db.hostname=dev1.domain.com
db.driver=com.mysql.jdbc.Driver
db.port=3306
db.url=jdbc:mysql://${db.hostname}:${db.port}/${db.database}

 

---

Back to top

Deploying once to multiple target environments

Name: Unified Deployment

Pattern: Create a single deployment script capable of running on different platforms and target environments.

Antipattern: Some may use a different deployment script for each target environment or even for a specific machine.

Although some deployment processes can run in certain environments, all processes should be capable of running in anytarget environment. For instance, the same Scripted Deployment is runin the development, test, stage, and production environments, but usinga different Externalized Configuration file. The externalizedconfiguration attributes are verified using the Template Verifier.

Figure 4 illustrates a single deployment script capable of deploying to multiple target environments:

Figure 4. Single deployment, multiple target environments

---

Back to top

But wait, there's more

Deployment is yet another aspect of software creation that lendsitself well to automation. Automated deployments reap the benefits of areliable, repeatable process: improved accuracy, speed, and control. Inthis article, I've covered eight patterns that can be effective forautomating software deployments. Part 2 will cover several morepatterns, including Remote Deployment, Disposable Containers,Deployment Test, and Environment Rollback.

 

Resources

Learn

• Software Configuration Management Patterns: (Stephen Bercuzk and Brad Appleton, Addison-Wesley Professional, 2003): The authors describe the Repository pattern and many other software configuration management patterns.
  
• Patterns of Deployment: HP Labs' SmartFrog wiki includes a catalog of many deployment patterns.
  
• Continuous Integration: Improving Software Quality and Reducing Risk (Paul Duvall, Steve Matyas, and Andrew Glover, Addison-Wesley Signature Series, 2007): Chapter 8, Continuous Deployment, demonstrates examples of incorporating deployment into an automated process.
  
• Pragmatic Project Automation: How to Build, Deploy, and Monitor Java Apps(Mike Clark, The Pragmatic Programmers, 2004): Enjoy pragmatic,automatic, unattended software production that's reliable and accurateevery time.
  
• Release It!: Design and Deploy Production-Ready Software(Michael T. Nygard, The Pragmatic Programmers, 2007): If you're adeveloper and don't want to be on call at 3 a.m. for the rest of yourworking life, this book will help.
  
• "Speed deployment with automation" (Paul Duvall, IBM developerWorks, January 2008): Leverage automation to move software through different environments quickly.
  
• Browse the technology bookstore for books on these and other technical topics.
  
• developerWorks Java technology zone: Hundreds of articles about every aspect of Java programming.
  

Get products and technologies

Ant: Download Ant and start building software in a predictable and repeatable manner.