That architecture still works fine, but an obvious improvement suggested itself since I proposed it. I've been finding that some jobs that I had encapsulated in programs are so ubiquitous that I could have just as well saved them as ado files. I had resisted doing that, because I tend to treat ado-files with a bit of reverence. Code, I think, is ado-file worthy if it is general enough and has proper online help.

Maybe I should ease up on that rule though. Maybe code that meets those requirements is actually package-worthy, ready to be distributed to the rest of the world, while the standards for ado-files should be more relaxed.

As long as your code does a job that's general within the scope of a given job, maybe it could go into an ado file that is available to that job and not others. There is an easy way to do that. Typing adopath in the Stata command line will list all the places where Stata will look for code for any commands you throw at it. Ado-files that you write go into the PERSONAL folder by default, but you can send them anywhere, as long as you tell Stata where to look for them.

Time for an example. I am working on a project where some do-files are built automatically (this is seldom necessary, by the way). Stata has some internal system limits (type help limits to see them all) and one of them is the restriction that a program cannot have more than 3,500 lines. This is very seldom a binding restriction. In fact, it should never even be an issue. But let's say you're stuck with legacy ways of writing code and adding to existing do-files, and you might run into this limit.

In such cases it may be interesting to have a routine for counting the lines in any do-file. Since that is done in the same way regardless of the content of the do-file, this job lends itself to being packaged into an ado-file. That ado file might look like this:

// counts lines in an ascii file (.do, .txt, .csv, etc)
// one argument: file name with path as needed
// (use full path for safety; spaces are OK)
capture prog drop lineCount
program lineCount, rclass

version 9.2
local filename `0'
tempname fh
local linenum=0
file open `fh' using "`filename'", read
file read `fh' line
while r(eof)==0 {
  local linenum=`linenum'+1
  file read `fh' line
}
file close `fh'
return local count `linenum'
end

So now, in your current do-file, you can do things such as

foreach client in `clients' {
  local this_client_file "`my_file_path'`client'_data_cleaning.do"
  lineCount `this_client_file'
  display "lines in `client'_data_cleaning.do: " `r(count)'
}

Now, under the old system, lineCount would have been a program defined in the Section 3 of either the current do-file or another do-file called by this one. But if I'm going to use it all the time and it never changes, making it a stand-alone ado-file instead makes sense. There are two steps for that. 

First, in the current project folder (let's say I defined is as the local `project_root') I set up a sub-folder called ado. Next, I set up a sub-folder called simply l, as in the first letter of lineCount. This may be overkill, but I like Stata's idea of grouping ado-files into subfolders by first letter. It's cleaner. Next, I save the program above into `project_root'/ado/l/ with the name lineCount.ado.

Second, I need to let Stata know where to look for it. That is as simple as adding

adopath + "`project_root'/ado"

at the top of my current do-file.

As time goes by and I find more jobs that could be handled this way, I can set aside their programs and save them as ado-files into sub-folders starting with their first letters. Stata only needs you to point it in the right direction. If it doesn't find your command in the ado folder, it will look for a sub-folder named after the first letter of your command.

Ado-files will keep my do-files less cluttered and will make it easier to both recycle old code and debug new one.

So, OK, you can widen the page, but should you? People read comfortably text that's about four inches wide, I would guess. Computers don't care, but code is meant to be read by people. The text you're reading used to sit in a box 540 pixels wide. The current width is 600 pixels. The difference was enough to make the problem go away. It would be nice to know that it's also not big enough to harm your reading convenience.

If you tell me that it is, I will revert to a narrower column, and start using all sorts of tricks to make Stata code fit inside it. Easiest, of course, would be to set #delimit ; . Then you don't care about end-of-line characters, because Stata will ignore them. But what if you're like me and you prefer #delimit cr?

You can use local macros as placeholders for whatever would otherwise result in long lines of code, as in

local 1 The quick brown fox
local 2 jumped over the lazy dog.
di "`1' `2'"

That's not too bad. I have used this workaround before, but now I'll make it an explicit policy: I will trade width for length in my code in future entries.

Maybe this should be a general standard of writing code. It might encourage diligent encapsulation. You don't want very long do-files, or at least you don't want very long programs, for all sorts of good reasons. So you already do spread large projects across multiple do-files, multiple programs or both, if you're really into that sort of thing. If you also force yourself to have short lines, your code will be all the better for it. What do you think?

My do-files start with a handful of header commands that I found useful at various times. They might look something like this:

clear
set more off
set type double
set mem 100m
pause on

This stuff varies a bit occasionally (I may set matsize or specify the version) but it's a bit like a cover sheet, in that it's almost always the same thing. I could have equally well put all of this in profile.do.

Next come the main sections of the do-file:

   Overview
   Globals
   Program definitions
   Program execution

The Overview section is all comments. It's got a structure of its own. It always consists of three parts. The first is just a list of the names of the programs defined in this do-file. The second describes what each program does, in one paragraph per program. The third describes which programs are called explicitly (because some programs on the list can be components of others) and in what order. The first part, the quick list, is useful for when you're fishing for code you might want to recycle. If you give your programs descriptive names, a quick look at the top of any do-file is usually enough to tell you if you're going to find useful stuff there.

The Globals section defines the macros such as file paths that will be used by more than one program. Since local macros are local to programs, anything that you mean to be shared across multiple programs must be a global. Having them all bundled here has another advantage. If you send your do-file to be run on another computer, all your file path changes are made in one place, once.

The Program definitions section does just what you might expect. Programs here can be stand-alone things that you call explicitly in the last section, or can be components, called implicitly. Defining such components is useful when you need to use the same code more than once. If that code is broken, you only need to fix it in one place.

You might want two types of comments with your program definitions. One is a header before the program define line (or, if you're cautious, before the capture program drop line) that tells you at least whether your program takes any arguments, lists them if yes, and tells you a little about each of them. You'd want to know, at a minimum, which ones are string and which are numeric. The other is a set of in-line comments, throughout the program definition, as needed. I find it useful to explain any local macros I declare with a couple of words at least.

The Program execution section does the actual work. It has consequences on disk and on screen.

I settled on this way of writing do-files after I took an online class on C++ at NC State. I treat Stata programs inside a do-file the way one would treat functions inside a .cpp file. My Program execution section is the equivalent of main(). In C++, function declarations (also known as prototypes) are mandatory and go at the top of the source file. My do-file equivalent for those is the Overview section. Except, of course, Overview is not mandatory at all. It consists solely of comments.

Having to submit to this sort of discipline might strike you as negating the benefits of Stata's easygoing nature. After all, if you pined for structure, you'd be programming in SAS, where it's mandatory. 

Well, there are two good very good reasons for structure: one is that your code must be portable across your team; another is that it must be readable two weeks later, when you will have forgotten all about it. But I still wouldn't want it imposed upon me by the design of the programming environment. Neither of those very good reasons overrides the importance of on-the-job fun. When you program, you want flow. You need to be free to write up things as they come to you. The programming environment should accommodate that. Only in the tidying-up stage, when your thinking's done and your problem's solved, should you need to worry about structure.

Programming environments that impose structure on you, as opposed to letting you volunteer it, result in beautiful code that takes a long time to write and robs you of most of the pleasure of solving the original problem. The latter might well cause you to do a mediocre job of it. When help like this also costs you more in licensing fees and specialized labor, that's just insult upon injury.

See, Stata is so flexible and easy to use that it's perverse in a way. Take for example the ability to generate new variables on the fly, either in interactive mode or wherever in your do-file fancy strikes. You would, of course, use Stata's generate command, or your favorite short version of it, as in

gen my_new_variable

You couldn't do this in SAS. First, there's no such thing as interactive mode, and second, you'd have to be in the data step. SAS will impose a modular structure on your code, and if that's not your style, too bad. You'll find it awkward but you'll either have to live with it, or hire a SAS programmer. There is, by the way, such a job title. I just put "SAS programmer" in the search box at simplyhired.com and got 518 hits. And what did "Stata programmer" get me? This:

Dang. We didn't find anything for you.
We couldn't find any jobs for "Stata programmer".
You're probably a good speller, but check the keyword terms you entered.
You can also try using some other keywords, or enter fewer words to expand your search.
It's also possible we made an error somewhere.
Sometimes computers are human too... just shinier.

This might look like a digression, but it's not. Where I was going is this: robust, friendly, and well-designed tools like Stata make programmers out of all of us. That's the upside. The downside is that while we might be fine social scientists, or epidemiologists, or what have you, we'll be almost sure to be pretty lousy programmers, because ease of use breeds incompetence.

Back to DRY: if you can do anything at any point in your do-file, you will not resist repeating yourself, because that's the most expedient thing to do sometimes. Worse, you will do the same thing but with slight variations, like spelling 2 in one place and `two' in another. That will produce ugly, unmaintainable code.

Yet DRY is as feasible in Stata as in the least user-friendly of its alternatives (I'm not saying that's SAS, by the way; machine code might be worse). You just need to set up and keep some sensible rules of your own for what's acceptable when writing a do-file. I will follow up with some suggestions over the next couple of days.

Stata will let you write your own ado-files and treat them as first-class citizens of your Stata install. But: (1) Stata is a very accomplished piece of software right out of the box, (2) it is updated scrupulously and as often as needed and (3) there are already thousands of user-created commands tried and true and ready to net install. So, it rarely warrants that sort of effort. Though this general assurance might not deter you from giving it a try, chances are that you're not an ado-file writer. You use do-files instead.

Do-files can get unwieldy with complicated projects. Yet the more complicated the job, the more useful your do-files are, because they organize your work and maintain your paper-trail. There is no reproducible research without do-files. This is so important that the fact that you can work with Stata interactively is sometimes argued to be too much of a good thing.

So you need do-files and you don't want them to be too big. One way to make that compromise is to use hierarchical do-files. A very short master do-file calls the others, each of which is just a list of commands that could have just as well been all in one file. Your master do-file might look something like this:

clear
set more off
pause on

cd c:/data/myproject
do collect_data.do   // insheets data from source text files
do estimate_model.do // runs my clever ML routine
do write_report.do   // that's right, from Stata to LaTeX


This would do the job, and inline comments as shown above can add intuitive appeal beyond what you get from simply using good do-file names. But if you need to run a do-file inside a loop, there is a certain cost to reading every single line of Stata code every time it needs to be executed. It would be nice if you could read it in once, and invoke it as needed. That's the use of program.

Usually, do-files are just text files full of Stata commands. Stata reads, interprets and executes these commands one by one. But any do-file can be turned into a program with three extra lines as follows:

capture program drop myProgram // this is optional, but you want it
program define myProgram

[...] // your old do-file goes here

end


This doesn't mean that all do-files should get the program treatment. But suppose that you have a do-file called clean_my_data.do and you use it for preparing 20 source files in the same way -- generate new variables, turn dates from yy/mm/dd into Stata's elapsed time format, etc. Without these three extra lines, your master do-file would call it like so:

forvalues i=1/20 {
drop _all
use mysourcefile`i'
do clean_my_data.do
}


That's fine, but it will be a bit slow. Every line in clean_my_data.do is read afresh in every cycle, output to screen, then executed, its result is output to screen, and so on. Declaring this do-file as a program allows Stata to read it once, keep it in mind, and execute it as many times as needed. You will, of course, choose some descriptive name for your program, say cleanMyData. Your master do-file will look like this:

do clean_my_data.do // Stata reads and memorizes your do-file

forvalues i=1/20 {
drop _all
use mysourcefile`i'
cleanMyData // Stata executes the commands inside your do-file
}


There is another advantage to declaring do-files as programs: clean logs. If you have Stata read the commands first and run them all at the same time from memory, then your logs will only contain the output of those commands, not the commands themselves. That will keep them small and readable.

The drawback is that programs are sticky: Stata memorizes them until it is either explicitly told to forget them, or is shut down. In the same instance of Stata you might possibly run two very different do-files that call two different programs declared at different times in the past under the same name. If in this instance of Stata you only read in one of them, it will be executed twice: once in its proper context, and another time in a totally wrong one. Stata will think that that's what you mean to do.

The capture prog drop myProgram line might help a bit. Obviously you can also capture prog drop _all, with the caveat that this might drop programs that you don't want dropped. Use a blend of the two variants as circumstances warrant. Stata can live with either. The best thing to do, of course, is to use very descriptive names for programs. They usually have the side effect of being specific, which will avoid such ambiguity in the first place.

You can set Notepad as your default do-file editor as follows: open Windows Explorer, say My Documents. The last item under the Tools menu is "Folder Options..." In the File Types tab there is a list of all the file extensions that your Windows XP installation knows of. Highlight the .do extension and click the Advanced button. There are two actions you can take with do-files: open and edit. Either can be edited. I know it's ambiguous, but when you're editing "edit" you're simply telling Windows what program to launch for editing. By default, Stata's installation instructs Windows to open your w[...]stata.exe and orders it to "doedit %1". You can change the Edit entry to "C:\WINDOWS\notepad.exe" "%1". That tells Windows that you want to edit your do-files with Notepad. Click "OK" and you're all set.