What I’ve been up to – A: Automation
Have a look at what’s been said about one of my projects:
Using the deployment automation tool, I just did a hot deploy of the [censored] training system for the first time, literally 5 minutes before a meeting at which I was going to demonstrate it, and the process was almost too intuitive and painless. Very impressive! –John Beaver (Software Developer)
Outstanding work … Great feedback, especially for a 1.0 release! The efficiencies you have built into the process and the tool provide material improvements and time savings for the [censored] program! Looking forward to v2.0 and integration with more applications. Super work! With my utmost gratitude, –Helen Baker (CCT Director)
This rocks—many thanks! I love automation. J Regards, –Tony Niderost (Sr. Director/GM)
GREAT WORK J!!! Thanks, –David Kuhl (VP, Corporate IT)
So what are they talking about you may ask?
I called it our Self Deploy app, and it did just that.
I’m a huge proponent of automation and simplification of complex processes– Even to the extent that I automate myself out of a job 🙂 I’m also a huge fan of Steve Jobs work. Let’s look at few quotes that really speak to me, of which I read in his biography:
Simplicity is the ultimate sophistication. –Steve Jobs
Simplicity isn’t just a visual style. It’s not just minimalism or the absence of clutter. It involves digging through the depth of the complexity. To be truly simple, you have to go really deep. –Steve Jobs
Steve made devices simpler by eliminating butons, software simpler by eliminating features, and interfaces simpler by eliminating options.
During one point I was on-call 4 weeks in a row; accountable 24/7 to ensure builds were released at the beck and call of anyone in Dev or QA departments. My motivation was to simplify this disjointed release process that spanned ~30 Linux/Windows servers & took up to an hr, so that anyone could run the deploy, and thereby freeing me to enjoy MY weekends. 🙂
Every deployment option was given a radio buton, tab, or drop down and a default value so that anyone could run the deploy process. The goal was a ‘single click’ deploy app and I did everything to reach that goal,.. even to the extent of securely encrypting the user’s credentials to a Windows store so it only needed to be typed on first run.
UPDATE: I contacted my old colleague and was told they are using the app to this day. That’s how I judge success!
Logging into Linux servers using ssh keys
The client I’m working for forces password updates every ~3 months, which means the 100 or so servers I’m logging into need to have their saved passwords updated in my SSH client software. I got really tired of this and so I decided to automate with a bash script in a secure way with ssh keys instead of passwords.
UPDATE: This post is primarily to share the bash script I wrote. For a more intuitive tutorial on my first steps 1-4, refer to something like this. The script I created helps propagate my public ssh key.
1. Install putty
2. in the program files folder of putty, start the app named puttygen
3. set key encryption to 2048. Click ‘generate’
4. save the private key to disk (copy the text of the public key and paste it into the script below for the value of publicSSHKey)
5. Copy the following into an executable file on your linux box. I called mine AddSSH.sh (chmod +x AddSSH.sh)
6. Execute the following as root… with sudo. (sudo AddSSH.sh)
#!/bin/bash
#
# AddSSH.sh
#
# Adds an SSH key to the authorized_keys file and allows system to accept SSH keys for login
# Requirements:
# 1. You must specify a valid user for the userid.
# 2. You must specify the public SSH key in publicSSHKey (Use putty's puttygen to create this").
# 3. Must run as rooooooot!
#
############################################################
# The user account you are enabling logins for
userid=""
# The public SSH key (you probably used puttygen to create)
publicSSHKey=""
# Source function library.
. /etc/rc.d/init.d/functions
clear
echo "Script name = $(basename "$0")"
echo -e "Script last modified. = \e[00;33m `date -r $0 +%c` \e[00m"
echo -e "\n\n"
# Ensure we are running as root
if [[ $UID != 0 ]]; then
echo "Please run this script as root (sudo su -):"
echo_failure
exit 1
fi
# Ensure the user specified in $userid exists
if [ ! -d /home/$1 ]; then
echo "the user specified in the script settings: ${1} , does not exist."
echo "You must edit this script and populate the userid variable with a valid user"
echo_failure
exit 1
fi
# Check for .ssh dir
dir1="/home/$userid/.ssh"
if [ ! -d $dir1 ]; then
echo ".ssh dir did not exist.. creating: $dir1"
echo_warning
mkdir $dir1
chmod 700 $dir1
chown $userid $dir1
fi
# Check for authorized_keys file
file1="/home/$userid/.ssh/authorized_keys"
if [ ! -e $file1 ]; then
echo "authorized_keys did not exist.. creating"
echo_warning
touch $file1
chmod 700 $file1
chown $userid $file1
fi
# append public key to authorized_keys
if grep -Fxq "$sshKey" $file1
then
echo_warning
echo "Key is already present in authorized_keys"
else
echo_success
echo "Appending SSH key"
echo $sshKey>>$file1
fi
# modify ssh login file to allow SSH key logins
sed -i 's/#RSAAuthentication/RSAAuthentication/g' /etc/ssh/sshd_config
sed -i 's/#AuthorizedKeysFile/AuthorizedKeysFile/g' /etc/ssh/sshd_config
echo_success
echo -e "User $userid should be able to login using SSH\n"
7. Now you need to specify the public ssh key from step four in your SSH client software. If your using Putty, you can find that here:
a. putty:
b: I’m using a Remote Desktop Manager (love this program and I highly recommend it to manage large numbers of Win/Linux boxes). I select the server, and hit cntrl-E to edit the session details. On the Connection tab I enter the details like you see below:
c. Which leads me into automation. If your using plink (another application that gets installed with Putty) or putty from the command line, just specify the location to the private key .ppk in qoutes with the -i switch.
Replace Icons in Program (notepad++)
I consider myself a ‘power user’ which really just means I have a ton of windows open in my taskbar throughout the day… Which makes it really hard to find the app you’re looking for if the icon that happens to be camoflauged by the Windows theme you’re running.
It seems like I’m always fishing for my most used app, Notepadd++, that I use for coding because the green/white icon seems to always be hiding. I Googled for a way to replace the icon and with a little hunting I came up on this thread, which not only provided a replacement icon, but also a hint as to the program I could use to make this happen:
Resource Hacker!
- I had to copy the notepad++.exe to my desktop before it would allow me to save (one of those Win7 things)
- I think I used the 101 resource to replace the icon (comment if I forgot)
Finished product:
Perforce auto login using md5 hash (p4v and cli)
A coworker of mine showed me a cool way of avoiding having to type your password each time you log into Perforce: An md5 of your password stored in an environment variable, or an .ini file if your using the command line.
1. in p4v, click Connection > Edit Current Workspace. Make a note of your ‘Root’. Mine is C:\depot\
2. Navigate to your ‘Root’ and create an file P4.ini
Mine looks like this:
# This is to tell the sd client what server and port to use
p4PORT=perforce:1985
P4HOST=a-box
P4USER=a-cool-guy
p4CLIENT=a-nskitch_hisWorkspace
P4PASSWD=(Get this from step 3)
2a: If you’re doing this for p4v, just create these as environment variables instead.
3. Go to one of the following websites and generate the md5 hash of your password and place it in your .ini file for the P4PASSWD value.
http://www.miraclesalad.com/webtools/md5.php
http://www.adamek.biz/md5-generator.php
- Note: The md5 hash MUST BE ALL CAPITALIZED. It took a bit of Perforce documentation reading to realize what my problem was.
- Note, instead of an .ini file, you can create an environment variable called P4PASSWD. I chose to use an .ini because at my work we connect to four different depots using different accounts. Because you can only have one environment variable, the .ini allows you to connect to as many depots as you want as long as the .ini file is placed in the root of your workspace. Here‘s a good description of the various ways to set these variables and their search order.
4. Add a System Environment variable named P4CONFIG with a value of p4.ini . This is how Perforce knows to look for our p4.ini file.
You can verify that perforce is finding your p4.ini file by typing from the command line: p4 set. You should see the output of this command matches the contents of your p4.ini file.
Result:
Each time you open Perforce, you will not be prompted for your password! Sweet!
Parsing Tomcat Catalina values with Grep/AWK
Blog Update
I’ve started a new job at a large travel company based in the Pacific North West. My job functions have shifted from software deployment on Windows, to release engineering in Linux for tomcat web servers. It’s a completely different world, but I want to continue blogging, so the theme of this blog will be changing to general release engineer topics in Linux.
Parsing Tomcat configuration settings:
CATALINA_BASE=`grep 'CATALINA_BASE=' $Tomcat | awk -F"=" '{print $2}'`serverXMLFile=$CATALINA_BASE/conf/server.xmlcatalinaPropertiesFile=$CATALINA_BASE/conf/catalina.properties
appBase=`grep 'appBase=' $serverXMLFile | awk -F"=" '{print $3}' | tr -d '"'`# ex: /app/webapps
C# DTF using locale .NET culture folder and .resources files for translations with MakeSFxCA.exe
Wow, that title is a mouthful.
One of the REALLY cool features of WiX DTF is that it uses MakeSFxCA.exe to create a self-extracting Dll that auto-magically bundles all your project dependencies into one DLL. When called from your installer at run-time, the files are extracted to your %temp%\MSI*.tmp\ folder and your DLL is called.
But, we had a little bit of trouble getting the out-of-the-box setup to work with .NET localization.
To provide a little background: We’re using an Installshield 2011 Basic MSI project with the Wix Toolkit’s C# Deployment Tools Foundation (DTF) to fire a custom action in the UI Sequence. The custom action dll displays a Winform of which we need to have localized for strings translations.
However, the out-of-the-box Visual Studio custom action project templates available to you after installing WiX do not properly deploy nested folder structures, which are a necessity for .NET localization.
To visualize, here’s the output of my project as compiled in Visual Studio… notice the .net culture sub-folder
..And here’s my %temp% folder it’s extracting to at runtime… notice that it’s not respecting the sub-folder with the .net culture naming syntax, and just bundling it all in the same folder. (LocaleTest.resources.dll). Without the .net culture sub-folder, the .resource file isn’t found and so it defaults back English at run-time.
After a lot of searching I stumbled upon this thread where a user had a similar question:
Is there a way to get MakeSfxCA include everything in my CA project ouput
directory recursively, maintaining folder structure? It seems to me this
would be the simplest way of ensuring all necessary dependencies are there
at runtime.
The user dug through the source code of MakeSfxCA (yay for open-source!) and found some comments in the code that pointed him to the correct syntax:
Reading the source for MakeSfxCA I found the GetPackFileMap function and
reading the description: “By default, all files will be placed in the root
of the cab. But inputs may optionally include an alternate inside-cab file
path before an equals sign.”So this implied to me that on the MakeSfxCA command line, rather than pass ”
$(TargetDir)EULAs\ja-JP\Eula.rtf” I could pass “EULAs\en-US\Eula.rtf=
$(TargetDir)EULAs\ja-JP\Eula.rtf”I beleive the log output of MakeSfxCA confirmed this:
Packaging files
MyInstallerCA.dll
CustomAction.config
Microsoft.Deployment.WindowsInstaller.dll
EULAs\ja-JP\Eula.rtf
So, the syntax is to specify the <Virtual File Path at run-time>=<File On Disk>. (Just like a .cab or .zip file)
The user opened a WiX change request to have this fixed along with an updated wix.ca.targets. However, Jason Ginchereau commented that it wasn’t such a great idea to make it the default behavior.
Here’s what we’ve done to overcome this limitation. We’ve added the following .bat file as a post-build step to our project that hard codes the .resource files for inclusion into our resulting xxxCA.dll. Yes, technically MakeSFxCA.exe is being called twice, but the end result is exactly what we needed.
%wix%SDK\MakeSfxCA.exe “$(TargetDir)$(TargetName).CA$(TargetExt)” “%wix%SDK\x86\SfxCA.dll” “$(TargetPath)” “$(TargetDir)Microsoft.Deployment.WindowsInstaller.dll” “$(TargetDir)Seagull.InstallWizard.dll”
REM Languages need to be added once they are available using the following syntax:
“es-ES\InstallWizard.resources.dll=$(TargetDir)es-ES\InstallWizard.resources.dll
After adding the batch file as a post-build step, our translations display property at run-time in our custom UI.
ISICE Validation: Documenting Custom Actions with ISCustomActionReference table
I found a few forum threads relating to Installshield’s ISICE10 validation errors I found interesting. Basically, you’re supposed to document what each custom action does in the ISCustomActionReference table.
However, it doesn’t appear these ISICE errors will stop you from achieving Windows Logo Testing, so I consider them more of a warning/bestpractice.
MichaelU gives his opinion on the relevance of such ISICE (Installshield Internal Consistency Evaluators) as it pertains to Window’s Logo testing:
Just for your information, all of the validators named ISICE~~ are custom, and while they are designed to assist with passing the Vista Logo tests, I don’t believe that Microsoft considers our ISICEs to be authoritative. Or, in short, if you run into a problem with a partiuclar ISICE validator, and your concern is acquiring the Vista Logo, it may be better to discuss it with those who run the Logo program. ISICE10 in particular reflects a particular method to document custom actions rather than the Logo goal of having them all documented.
From user DebbieL, a Technical Writer at Installshield:
The errors about specific custom actions not being documented are validation errors for validator ISICE10. Here’s the information in the documentation about this validator:
The intended behavior of each custom action must be documented for the Windows Vista Quality Program. This is especially helpful if system administrators deploy your product to enterprise environments; they sometimes need to know what the custom actions do.
ISICE10 verifies that each custom action in your installation is documented by validating that each entry in the CustomAction table has a corresponding ISCustomActionReference table entry.
Corrective Action
To resolve this validation error, open the Custom Actions view, select the custom action that is mentioned in the error message, and use the Help File Path setting to specify a path to the document that describes the behavior of the custom action. When you specify a value in the Help File Path setting, InstallShield adds a row for that custom action in the ISCustomActionReference table if one has not already been created.Note that if the custom action is a merge module that is consumed in your installation project, specify the path in the Custom Actions view of the merge module project and then rebuild the merge module.Here is some additional information from the help topic called “Documenting the Behavior of Custom Actions”:
To document the behavior of a custom action in your project:
1. Create a file that describes the intended behavior of the custom action. It does not matter what type of file that you use. Note that each custom action should have its own document.
2. In the View List, under Behavior and Logic, click Custom Actions.
3. In the Custom Actions explorer, click the action that you are documenting.
4. For the Help File Path setting, click the ellipsis button (…) to browse to the file that describes the behavior of the custom action. The file should be a text-based file such as a .txt, .htm, or .rtf file.TIP: You can specify whether InstallShield should stream the contents of each of the custom action help files into the .msi file at build time. For more information, see the description of the Include Custom Action Help setting for a product configuration in the Releases view.
In another thread, a user explains that he followed the above directions but was still receiving the error. Here’s the additional setting he needed:
After struggling […], I found a cause of this problem.
I did not set the Include Custom Action Help setting to “Yes” for the Product Configuration.To resolve the ISICE10 error, I did the following operations:
In the View List, under Media, click Releases.
Select the product configuration that has the validation problem.
Check the value of Include Custom Action
Help and set the value to “Yes” if it is “No”.
Major Upgrade using Product Code
Caution: This is NOT best practice! But when your back is against a wall, you need a way out. This was my way out/solution.
We recently got ourselves stuck up a creek without a paddle, so I had to fabricate a paddle and here’s how I did it. This code will comb the registry for the existence of a list of supplied product codes and attempt to uninstall the first one found in the UI Sequence using msiexec /x.
The back story of why we needed this isn’t really important, but I’ll share the synopsis. We have 2 types of installers and the second type is more or less a trial that has ~25 flavors. The two types of installers can co-exists on the same machine (unique upgrade codes). Well for one reason or another it was decided that this should no longer be the case and for manageability we would combine the two sets of flavors into one upgrade code so only one product can be installed on a machine at a time. This was done and we performed a release. Months later the Marketing dept caught wind of this and wanted the old behavior back to allow for both the flavors to co-exist on the machine at a time. (2 different Upgrade Codes) THIS WAS A PROBLEM. Now that the 2 flavors of our software share the same upgrade code, they are identical as far as MSI is concerned and there is no way to delineate between the two in order to perform major upgrades. — So this is when I came up with the following solution we’ve had in place for awhile. (Sorry, .. long ‘synopsis’)
MSI can only perform major upgrades using Upgrade Codes. There’s no built in mechanism for using a Product Code to uninstall a product. Here’s how I’m targeting specific Product Codes of our past release (25 individual products) and manually uninstalling the first one found using msiexec.exe /x during the UI Sequence. (Can’t be done during the Execute sequence because msi-chaining is not permitted). The following is Installscript.
//===========================================================================
//
// Function: UninstallIfProductCode()
//
// Description: Removes specific product codes. Removes one and then returns.
// Note: (There would only ever be one on the system.)
//===========================================================================
export prototype VOID UninstallIfProductCode(HWND);
function VOID UninstallIfProductCode(hMSI)
STRING szProductNumber,szCompressedGuid,szKey;
LIST productCodes;
NUMBER nvItem,nResult;
begin
productCodes = ListCreate(STRINGLIST);
ListAddString(productCodes, "{EAFD321F-7441-49F7-845F-162AFE9A9E89}", AFTER);
ListAddString(productCodes, "{15219BB7-218A-4CA9-B904-E9730A5AE831}", AFTER);
ListAddString(productCodes, "{831E34B1-844F-485D-95F6-0B4324D30F3F}", AFTER);
ListAddString(productCodes, "{2ECC9820-1A29-4860-8A0B-BD1CE320B4B2}", AFTER);
ListAddString(productCodes, "{A8CA60A2-5C80-4862-B539-72A0FE164631}", AFTER);
ListAddString(productCodes, "{BF0ADDB3-FBF4-45F3-8862-739B43D10F09}", AFTER);
ListAddString(productCodes, "{240047B1-1ED9-42DB-AC97-547D9C7B639C}", AFTER);
ListAddString(productCodes, "{02161687-5FB6-4123-9786-C6D41DA34B8E}", AFTER);
ListAddString(productCodes, "{33CAB057-95D1-40EC-A0C3-C116E4E3A58D}", AFTER);
ListAddString(productCodes, "{E3028A13-E396-47AA-ACB6-DE4F51954F0C}", AFTER);
ListAddString(productCodes, "{DCD725C7-BB93-49FF-AFA3-BD2FA96EC048}", AFTER);
ListAddString(productCodes, "{57CB7704-9BAA-407C-B3A8-EA8184CABF66}", AFTER);
ListAddString(productCodes, "{F67AA111-D082-4187-932B-9186DA4E2F6B}", AFTER);
ListAddString(productCodes, "{AEC516A4-6929-43B0-A4FD-3E7F9BFBBCBA}", AFTER);
ListAddString(productCodes, "{BB9B1A98-7A23-400D-B8BD-CBD7C3D2B839}", AFTER);
ListAddString(productCodes, "{E1BB77D9-C487-4AB1-BA9C-7CD752A63E5E}", AFTER);
ListAddString(productCodes, "{08C40B1B-B1D2-4DD2-87E1-677AAE040CBF}", AFTER);
ListAddString(productCodes, "{FB485301-53C7-4011-9256-3E7F9C92AC5F}", AFTER);
ListAddString(productCodes, "{9CAAE3D3-17D8-44CD-8659-62E29E64C64B}", AFTER);
ListAddString(productCodes, "{9CE9C04A-ADD6-45C6-90E3-4E56EDB2CB3C}", AFTER);
ListAddString(productCodes, "{724FCA6E-07C0-4D18-95A2-04CE0450A912}", AFTER);
ListAddString(productCodes, "{35DC018B-4DCF-42B9-B898-718EDEACAFD8}", AFTER);
ListAddString(productCodes, "{AA8B768D-5483-4D6C-8E73-12983D727C59}", AFTER);
ListAddString(productCodes, "{59C22288-4019-4179-B303-D7274F23457A}", AFTER);
nResult = ListGetFirstString(productCodes, szProductNumber);
while (nResult != END_OF_LIST)
szCompressedGuid = CompressedGUID(szProductNumber);
RegDBSetDefaultRoot(HKEY_CLASSES_ROOT);
REGDB_OPTIONS = REGDB_OPTIONS | REGDB_OPTION_WOW64_64KEY; //let's be 64bit aware
if(RegDBKeyExist(szKey) > 0) then
LaunchAppAndWait("msiexec","/x " + szProductNumber + " /qb /passive", WAIT);
goto productFound:
endif;
nResult = ListGetNextString (productCodes, szProductNumber);
endwhile;
productFound:
ListDestroy(productCodes);
end;
Note: The code for the CompressedGuid() function can be found in one of my earlier blog posts: Installscript to Transform GUID into ‘Compressed GUID’.
Like I mentioned, we’ve had this in place for a couple releases without any reported problems.
TweetDeck and Adobe AIR constant annoying updates.
This isn’t really related to software deployment. But, because twitter is a tool of my BuildMaestro online persona, this is where I’m gonna rant! It seems like every other time I start TweetDeck I get a VERY annoying popup notifying me there’s an update for Adobe AIR (what Tweet Deck, a twitter client, runs on). I usually hit the ‘update later’ button, but this just kicks the can down the road (sort of like QE and the impending sovereign debt crisis) and the nag screen just waits for the next time I use the app.
I’m but a mere user who has no idea what your update does, and I’m certanly the last person to know if this particular update is critical or just an agile development 2 week release cycle. I say, if you need to update, at least give me the option to have it silently done in the background. But Alas, digging around through the settings and Googling provided me with no hints.
Then I stumbled on this:
Q: Is it possible to disable Adobe AIR auto-updates?
A: Yes. You can turn the AIR update feature on or off via the AIR SettingsManager application. The AIR SettingsManager application is available for download athttp://airdownload.adobe.com/air/applications/SettingsManager/SettingsManager.air.
Adobe highly recommends that, if you disable automatic updates, you manually apply all runtime updates as soon as possible.
Yes! No more nagging. I’ll update when there’s a critical exploit in the news or my twitter client falls to it’s knees with version obsolescence. I think it’s kind of funny that you have to download a separate Adobe AIR application just to change a setting!
Another very annoying update notifier is the Java VM client. Same applies here and I was able to also turn this off via the Control Panel java applet settings.
.NET Version Detector 2007
I just wanted to share a recent find of mine (where have a I been?) that’s a pretty handy tool to show which versions of the .NET Framework are installed on a machine. Sure, there are plenty of ways to find out which version is on the machine: Add/Remove programs and updates, Server Manager, Registry keys, the version of fusion.dll.. etc etc.. But I found a tiny tool that simplifies it. I put this on a network drive and I can run it remotely from any machine.
Link to download: .NET Version Detector 2007
the Build Maestro 