Release Engineering 101

How to make sure you can reproduce your codebase, and tool-chain, and not get burned by missing dependencies.

These are some guidelines that I’ve learned over the last 40 years in the software business. For the last 18 years of my career, I was primarily doing Release Engineering–taking code from developers and getting it into production.

1. Always document the actual source URL and any other information needed to access the source code or binaries. Include credentials or the location for the credentials (for example, see KeePass.)

Why? You’ll want to work on other projects and not be the information provider forever.

General Release Directory Structures

TOC

• Preface
• Directory Levels
  • <FunctionalDir>/
  • <FunctionalDir>/trunk/
  • <FunctionalDir>/<BranchName>/
  • <CategoryDir>/
  • <WebSiteDirName>/
  • <CodeDirName>/
• Repository or Site Types
  • Version Directory
  • Release Repository
  • InternalWebSite
  • InternalWikiSite
  • README Files
• Directory Examples
  • Version Repository
  • Release Repository
  • Internal Web Site
  • Internal Wiki Site

---

Preface

This document is to be used as a guideline for organizing and naming document folders and directory trees, which are shared by many people. This is a structure that I have evolved over 15 years while working at 6 different companies. I figure it would be on major version “8.0”, if it were a product. In the last two companies, I haven’t had to make any major structural changes. Of course, only about 10% of the top levels get used, but they are there if the categories start to get “too deep.”

A number of goals are embedded in this structure:

• Avoid the 90-degree collision of mixed structures. For example, a structure could start with “projects”, then sub-folders are used for “functional” areas, marketing, engineering, etc. While another structure starts with functional areas, then the project is repeated across each of the areas. Either structure could work, but using both will lead to confusion. The “project” structure is what most people start with, but it has a major flaw. Often documents in the project need to be shared with external partners, but functional groups will want to keep some documents private to their group. So the project files start to be copied to alternate structures, because opening up the top project directory, to external partners, reveals too much information. In other words, I’ve noticed that the sensitivity of information tends to be grouped stronger by functional boundaries, than by projects. Marketing will partner with an external group, but engineering wants to keep its code private. Engineering may outsource the work, and marketing will want the business plans kept secret. This is a problem because the whole point of a repository is to share information. So the top directories for this template are by functional areas, not project.

• Avoid using different names for the same thing. For example, if you have configuration files, these can be collected in a directory named: etc, or config, but don’t use both names. Pick one and be consistent within the functional area or “category”. This document lists some recommended names for different categories. Before making up a name, look around the other project areas and notice the naming patterns used by other projects. Don’t assume that the oldest or even the newest names are the preferred names–other engineers could have just not cared enough to make things consistent with other code. If you abbreviate a name, then always use the abbreviation.

• Clean-code refactoring tip: if you see an inconsistent naming pattern (either not self-consistent or not company style consistent), then fix it! Your IDE should make it easy to change names across multiple files. If not, learn to use find, grep, sed, perl, etc. A professional engineer should be able to make global changes across thousands of files with confidence in their tools.

• The top-level directories are “fixed”, and sibling directories, at the same level, categorize the same content. This makes it easy to write scripts for archiving and publishing. It also makes it easier to create and include paths in software code, because they are more “regular”. It is also easy to add new libraries, with little or no changes to the build scripts.

• The structure of similar content is grouped together.

• The “branch level” for software can be placed at almost any fixed level. But keep that level consistent. Do not branch at other levels. (Note: this part is not relevant for “git” repositories.) Notice the branches are on the same level as trunk. Creating a “branches/” dir. level is unnecessary, because there should not be that many “active” branches. Inactive branches are deleted to keep the upper levels clean. Of course, the deleted branches are still available in the repository. With subversion and other modern version management tools, there also isn’t much need for a “tags/” directory. If you want to keep track of the source that goes with particular versions, an easier way is to just create a “Version” file that holds the version tag name and the repository path (branch) and id for that version. Using this style of branching and tagging, will keep the workspaces “clean” and focused on “active” code, while still having the flexibility to quickly get any desired branch and version. The best branch points are: just under each functional area or just under each category area. When branching, it is not necessary to include all of the content. Only the content that will be changed or saved, needs to be “copied” to the branch. The branch level is mainly determined by what parts a company normally includes in each release and on the dependencies of the parts. Of course, the branch level is not needed for the non-versioned areas, like a wiki tool, or a shared directory.

---

Directory Levels

This is the most important part. Each level has a “meaning.”

Simple Made Easy

Summary

Rich Hickey emphasizes simplicity’s virtues over easiness’, showing that while many choose easiness they may end up with complexity, and the better way is to choose simple first, then easy, if it is also simple.

This shows the classic problem with valuing writing code fast, with no regard to the 80% of the time that will be needed to maintain the code. Ignoring this leads to legacy code, periodic full rewrites of a codebase, and continually rewriting and debugging the same algorithms.

http://www.infoq.com/presentations/Simple-Made-Easy This is a good version of his talk.

Included here is a YouTube version of his talk. This one isn’t as good, because the slides are small. I also think the other version (above) was a better presentation.

Clean Code

I’ve been learning Google Apps Script, which is basically JavaScript. I’ve also been learning Test Driven Development. Sure, let’s learn three new things at the same time!

I was motivated to try TDD by this video:

In this video, he demonstrates a TDD Kata for creating a function that will find prime factors of numbers. I want to write code like that!

My google drive was getting cluttered with files that were named by others, who had a (bad) habit of using spaces and lots of other special characters in the file names. (Usually, this is because people try to encode too much information into a file name. But that is another topic.) I would download, version, and process some files, then upload them. Spaces and special characters in the file names messed up my ability to write “simple” bash scripts. So how about a renaming tool to normalize the file names? I.e., convert all the non-alphanumeric characters to ‘_’ (also allow ‘.’ and ‘-‘).

For the tests, I needed to create and recreate test folders and files in Google Drive. It was really tedious to do manually. So let’s automate it. I wrote a routine that would parse a nested array structure that represented the folders and files.

This was the object that I created. (This is the first version. I now have a version where you can optionally pass the array as a parameter.)

The Great Software Stagnation

Regarding:
The Great Software Stagnation

My comment:

Open Source and “free” software can be really confusing. For example see Richard Stallman’s points at: Open source misses the point

I think the analogy to music and movie is getting closer to the real issue: copyrights. When and what can be copied? When $ gets involved, the time gets extended to a very long time. In the U.S.A. 70 years after the author’s death. What about copyrights owned by corporations? When do they “die”?

Software has another subtly: the source code vs the compiled executable code. Here an automobile analogy is often used. Only selling or sharing the executable, but not the source code, is equivalent to selling a car with the hood locked so that a mechanic cannot repair it. If a mechanic were to completely copy the engine under the hood, then sure, they would be in violation of “copyright” or “patent” laws. But replacing broken components with the same or better components–that is OK. Repairing a defect in software should also be possible, without the permission of the manufacturer. Unfortunately, code is very easy to copy vs copying a physical device.

I think the solution to the “source code” issue is something related to the intent of copyright and patent laws: they exist to create a “temporary” (short-term) monopoly for the creators (not their heirs) so that they will be rewarded for releasing their works into the public domain. Otherwise, software or recipes for how things are made could be lost.

‘Not My Problem’: A Big Problem for DevOps Teams

Reposted from: ‘Not My Problem’: A Big Problem for DevOps Teams – DevOps.com

Not my problem (NMP) — (n)

1. a statement, or position, of apathy expressed by those who perceive they are external and unaffected by a negative predicament. While sometimes warranted, it is typically uttered by those who perceive themselves as powerless; can’t be bothered; are too lazy, or are selfish non-contributing leeches. See also “complete cop out.”

2. an attitude that will stymie attempts to implement DevOps in your organization and will thwart success

3. (archaic) Actually not your problem

While perhaps it’s become more frequent in our culture of relative indifference, some of our oldest stories bear witness to how timeless the phenomenon “not my problem” is. For example, in the biblical story of Cain murdering Abel, God asks Cain afterward, “Hey Cain, I can’t find Able. Do you know where he is?” Cain responds famously, “Am I my brother’s keeper? (NMP)” It crosses country borders and language barriers, too: A transliteration of the same sentiment in Polish is, “Not my circus, not my monkeys.”

A comment left on Slashdot. – Development Chaos Theory

(Reblogged from A comment left on Slashdot. – Development Chaos Theory)

The problem is that our industry, unlike every other single industry except acting and modeling (and note neither are known for “intelligence”) worship at the altar of youth. I don’t know the number of people I’ve encountered who tell me that by being older, my experience is worthless since all the stuff I’ve learned has become obsolete.

This, despite the fact that the dominant operating systems used in most systems is based on an operating system that is nearly 50 years old, the “new” features being added to many “modern” languages are really concepts from languages that are between 50 and 60 years old or older, and most of the concepts we bandy about as cutting edge were developed from 20 to 50 years ago.

It also doesn’t help that the youth whose accomplishments we worship usually get concepts wrong. I don’t know the number of times I’ve seen someone claim code was refactored along some new-fangled “improvement” over an “outdated” design pattern who wrote objects that bare no resemblance to the pattern they claim to be following. (In the case above, the classes they used included “modules” and “models”, neither which are part of the VIPER backronym.) And when I indicate that the “massive view controller” problem often represents a misunderstanding as to what constitutes a model and what constitutes a view, I’m told that I have no idea what I’m talking about–despite having more experience than the critic has been alive, and despite graduating from Caltech–meaning I’m probably not a complete idiot.)

Last In – First Out: Ad-Hoc Verses Structured System Management

This is a repost. This is an excellent list of all the operation areas that need to be well managed. The theme is: with automation, templates, and monitoring. Not with manual fiddling until it works. See below, for the link to the full post.

Structured system management is a concept that covers the fundamentals of building, securing, deploying, monitoring, logging, alerting, and documenting networks, servers and applications. Structured system management implies that you have those fundamentals in place, you execute them consistently, and you know all cases where you are inconsistent. The converse of structured system management is what I call ad hoc system management, where every system has it own plan, undocumented and inconsistent, and you don’t know how inconsistent they are, because you’ve never looked.

You know you have structured system management when:

VPro Intel Chips Contain Back-Door Processor

New Intel-Based PC’s Permanently Hackable

So you think no one can access your data because your computer is turned off. Heck it’s more than turned off, you even took the main hard drive out, and only the backup disk is inside. There is no operating system installed at all. So you KNOW you are safe.

Frank from across the street is an alternative operating systems hobbyist, and he has tons of computers. He has Free BSD on a couple, his own compilation of Linux on another, a Mac for the wife, and even has Solaris on yet another. Frank knows systems security, so he cannot be hacked… or so he thinks.

The government does not like Frank much, because they LOVE to look at everything. Privacy is a crime don’t you know, and it looks like Frank’s luck with privacy is about to run out.

The new Intel Core vPro processors contain a new remote access feature which allows 100 percent remote access to a PC 100 percent of the time, even if the computer is turned off. Core vPro processors contain a second physical processor embedded within the main processor which has it’s own operating system embedded on the chip itself. As long as the power supply is available and in working condition, it can be woken up by the Core vPro processor, which runs on the system’s phantom power and is able to quietly turn individual hardware components on and access anything on them.

Intel’s new Anti Theft 3.0, which put 3g connectivity into every Intel CPU after the Sandy Bridge version of the I3/5/7 processors. Users do not get to know about that 3g connection, but it IS there.

To read more: New Intel Chips Contain Back-Door Processor, Hackable Even When Computer is Turned Off | PopularResistance.Org

http://www.webcitation.org/6iaL5L0PS

No, you are not doing DevOps (…and nor am I)

Good rant and insights.

A subset of #2, keep the operators and sysadmins, but make them subordinate to developers, who will define new production architectures that can be changed just as fast as their continuous development environments. They get to play around in production, ignoring many years of experience in managing large scale infrastructures. Oh they will eventually get things stable, and working well, with a custom mix of tools and patches. However the next batch of developers will wonder “what it going on?” and they will start rewriting things until they understand it. Of course business will not mind unstable web sites, down time, and lots of engineers working to continuously reinvent things.

A sysadmin's logbook

A word of caution

This post is addressed to all those people who think they know what DevOps means, but they don’t. If you don’t recognize yourself in this post, then maybe it’s not for you. If you recognize yourself, then beware: you’re going to be insulted: read at your own risk and don’t bother asking for apologies.

View original post 1,170 more words