Unfortunately, include does not work well with programs. You can invoke it inside a program but include itself cannot be compiled, so the do-file it refers to will have to be interpreted all over again every time you call that program: see help include for the original reference. This negates the speed advantage that you expect when you package your Stata work into programs as opposed to simple do-files. It does not negate all the other advantages -- such as better code portability and the possibility of unit testing -- but still, speed matters. My own solution is to define locals inside an r-class program. Then other programs that need some of these locals call that program, and recover only the locals they need. Below is one example, worked first with include and then with my proposed alternative:

// locals.do starts here
local file_path "c:/data/my_path/"
local file_name "file_name.dta"
local my_file    "`file_path'`file_name'"
// and ends here.

OK, now let's put these locals to work:

// work.do starts here:
include locals.do
use "`my_file'"

Alright. This is not bad and it does the job that your locals are neatly separated from the rest of the work. If you use do-files to split a big job into smaller pieces, this is all you need. If, on the other hand, you want to chop the job into pieces smaller still, and encapsulate those pieces into programs, then this is inadequate, as explained above.

The workaround might look like a bit more overhead, but then that's the usual cost of writing programs instead of plain do-files. Some of us, sometimes, find that overhead acceptable:

capture prog drop defineMyLocals
program defineMyLocals, rclass

local file_path "c:/data/my_path/"
local file_name "file_name.dta"
local my_file    "`file_path'`file_name'"

local things "file_path file_name my_file"
foreach thing in `things' {
   return local `thing' ``thing''
}

end

And now let's recover the local `my_file':

defineMyLocals
local my_file `r(my_file)'
use "`my_file'"

That's it. Your locals.do consists of the definition of the program defineMyLocals. The overhead is obvious. First, adding new locals inside this definition requires the extra step of adding their names to the things list, so that defineMyLocals can return them. Second, using defineMyLocals is also slightly more complicated than a simple include, because you need to recover the locals you need explicitly, via `r()'.

That said, now your locals are defined inside a program, and you can move them around faster. If you're like me and you favor programs over do-files and this isn't your solution, then I'd be curious to see your preferred alternative.

See, Stata locals are just strings that come with a reasonable kind of automatic type conversion designed to make your life easier: if your string consists of numbers only, it will be evaluated. For example, local a=1 and local a 1 have the same effect: display `a' will return 1, the number, in either case.

But there is a subtle difference between an empty local with a name and a missing local. Stata, in a pinch, will identify either as a missing value, and that may work for you most of the time. But only the former can be called in a numeric expression that relies on automatic type conversion. The latter will give you a "missing name" error. Below is a simple example:

capture prog drop myLocal
prog def myLocal, rclass

local a
return local myresult `a'

end

myLocal
local a `r(myresult)' // note (1): this will work fine

if `a'==. {
  di "local a is missing"
}
else {
  di `a'
}

The code above will abort with the error message "nothing found where name expected r(198);", but that error message won't come at the definition of local a commented with note (1). That definition will work as if everything were fine: it simply returns a missing value. It's when you're calling the subsequent if-condition that you run into trouble.

The solution is to make sure that myLocal is returning what you mean it to. If you want it to return a missing numeric value, this is the way to go:

capture prog drop myLocal
prog def myLocal, rclass

local a=.
return local myresult `a'

end

myLocal
local a `r(myresult)'

if `a'==. {
  di "local a is missing"
}
else {
  di `a'
}

If instead you want to return a missing string, you need to make sure that your if-condition is a string condition:

capture prog drop myLocal
prog def myLocal, rclass

local a
return local myresult `a'

end

myLocal
local a `r(myresult)'

if "`a'"=="" {
  di "local a is missing"
}
else {
  di `a'
}

This is not as pointless as it looks. You seldom want programs to return a missing value as a result, of course. But a missing value can sometimes be a legitimate member of the set of possible results. When you run into one, you need a way to handle it.

So, OK, you can widen the page, but should you? People read comfortably text that's about four inches wide, I would guess. Computers don't care, but code is meant to be read by people. The text you're reading used to sit in a box 540 pixels wide. The current width is 600 pixels. The difference was enough to make the problem go away. It would be nice to know that it's also not big enough to harm your reading convenience.

If you tell me that it is, I will revert to a narrower column, and start using all sorts of tricks to make Stata code fit inside it. Easiest, of course, would be to set #delimit ; . Then you don't care about end-of-line characters, because Stata will ignore them. But what if you're like me and you prefer #delimit cr?

You can use local macros as placeholders for whatever would otherwise result in long lines of code, as in

local 1 The quick brown fox
local 2 jumped over the lazy dog.
di "`1' `2'"

That's not too bad. I have used this workaround before, but now I'll make it an explicit policy: I will trade width for length in my code in future entries.

Maybe this should be a general standard of writing code. It might encourage diligent encapsulation. You don't want very long do-files, or at least you don't want very long programs, for all sorts of good reasons. So you already do spread large projects across multiple do-files, multiple programs or both, if you're really into that sort of thing. If you also force yourself to have short lines, your code will be all the better for it. What do you think?

If you write code using formally defined programs, as I suggested in my previous post, then you may have already run into the problem of making your programs pass each other various results. One way is to have the programs store their results as global macros, which then are available to any subsequent programs within the same Stata instance. This is inelegant and risky, because your globals might stick around longer than you want them (OK, they won't survive shutting Stata down and restarting it, if you care to do that all the time).

The other way of enabling programs to use each other's results is rclass. Defining your program this way enables it to return r() results, which are then available to subsequent programs, no globals needed. An example is below:

// my first program defines the answer
capture prog drop theAnswer
program define theAnswer, rclass

return scalar myanswer=42

end

// my second program calls the first
// and uses its r() result.
capture prog drop theAnswerIs
program define theAnswerIs

theAnswer
local answer=r(myanswer)
di "The answer is `answer'."

end

Now call the second program in Stata:

theAnswerIs

Congratulations. You now have the answer to the Ultimate Question.

Packaging a set of commands inside a program, to be read once and then invoked as many times as they are needed, is nice enough. But what if you need to make those instructions apply to different variables in different instances?

For example, I have a set of parameter estimates for a survival model. I use them to calculate the lambda part of the hazard function under various scenarios for one variable of interest. You can do this immediately after estimating your hazard model, of course, using predict with appropriate modifiers, but you don't want to re-estimate the model every time you need to use the parameters. Besides, you may have to use one data set for estimating the model (say historical data) and other data sets (say current data) for making the predictions of interest.

So let's say that you have your parameters saved in a matrix, and declare a program, called e.g. getLambda, to pair them with the corresponding regressors as many times as you need to. But every time you call getLambda, instead of the first regressor you need to use a variable that is some sort of transformation of it, and you saved that variable with a different name to make it absolutely clear that this is not the actual regressor. It would be nice if you could pass that variable as an argument to getLambda, in effect telling Stata "I want to calculate lambda with this particular variable at the front of the regressor list this time".

The program getLambda might look something like this:


capture prog drop getLambda
prog def getLambda

local thiscase `1'
local myregressors "`thiscase' var1 var2 var3 1"
local regressorcount: list sizeof myregressors

forvalues i=1/`regressorcount' {
   local thisreg: word `i' of `myregressors'
   gen param_`i'=beta[`i',1]*`thisreg'
}
egen lambda_`thiscase'=rsum(param_1-param_`regressorcount')
drop p_*

end

I forced the last element of `myregressors' to be 1 because I am assuming that the constant is the last parameter saved in your parameter matrix. Notice the local `1' in the first line after the program definition. It is Stata's name for the first argument of your program. In our case, that argument is simply the name of the variable that you want in the first position in the regressor list. So the call to getLambda would be


getLambda foo

This would make Stata calculate the lambda with the variable foo at the front of the regressor list. If later you want the variable bar there instead, you just call

getLambda bar

Arguments can be strings, numbers, whatever. As long as your function is strictly for internal use and you know what you're using it for, you don't need to program the fancy stuff -- like error codes or help files. That, at least, is what I found. When I get to the point where I need those extras, I'll learn how to do them and then blog about it here.

You may have noticed the apparently superfluous use of the list sizeof extended macro function. After all, it's plain enough that the word count of the "myregressors" local is 4. Well, sometimes you have more regressors than that. Do you care to count them by hand? I don't.

There are all sorts of very good reasons for leaving to Stata as much of the grunt work as possible. The most obvious of them is that code written this way is more portable across different projects. But the honest truth is that I really like those list functions. I've been finding all sorts of uses for them lately. I am going to have a blog post dedicated to them sometime soon.

Stata will let you write your own ado-files and treat them as first-class citizens of your Stata install. But: (1) Stata is a very accomplished piece of software right out of the box, (2) it is updated scrupulously and as often as needed and (3) there are already thousands of user-created commands tried and true and ready to net install. So, it rarely warrants that sort of effort. Though this general assurance might not deter you from giving it a try, chances are that you're not an ado-file writer. You use do-files instead.

Do-files can get unwieldy with complicated projects. Yet the more complicated the job, the more useful your do-files are, because they organize your work and maintain your paper-trail. There is no reproducible research without do-files. This is so important that the fact that you can work with Stata interactively is sometimes argued to be too much of a good thing.

So you need do-files and you don't want them to be too big. One way to make that compromise is to use hierarchical do-files. A very short master do-file calls the others, each of which is just a list of commands that could have just as well been all in one file. Your master do-file might look something like this:

clear
set more off
pause on

cd c:/data/myproject
do collect_data.do   // insheets data from source text files
do estimate_model.do // runs my clever ML routine
do write_report.do   // that's right, from Stata to LaTeX


This would do the job, and inline comments as shown above can add intuitive appeal beyond what you get from simply using good do-file names. But if you need to run a do-file inside a loop, there is a certain cost to reading every single line of Stata code every time it needs to be executed. It would be nice if you could read it in once, and invoke it as needed. That's the use of program.

Usually, do-files are just text files full of Stata commands. Stata reads, interprets and executes these commands one by one. But any do-file can be turned into a program with three extra lines as follows:

capture program drop myProgram // this is optional, but you want it
program define myProgram

[...] // your old do-file goes here

end


This doesn't mean that all do-files should get the program treatment. But suppose that you have a do-file called clean_my_data.do and you use it for preparing 20 source files in the same way -- generate new variables, turn dates from yy/mm/dd into Stata's elapsed time format, etc. Without these three extra lines, your master do-file would call it like so:

forvalues i=1/20 {
drop _all
use mysourcefile`i'
do clean_my_data.do
}


That's fine, but it will be a bit slow. Every line in clean_my_data.do is read afresh in every cycle, output to screen, then executed, its result is output to screen, and so on. Declaring this do-file as a program allows Stata to read it once, keep it in mind, and execute it as many times as needed. You will, of course, choose some descriptive name for your program, say cleanMyData. Your master do-file will look like this:

do clean_my_data.do // Stata reads and memorizes your do-file

forvalues i=1/20 {
drop _all
use mysourcefile`i'
cleanMyData // Stata executes the commands inside your do-file
}


There is another advantage to declaring do-files as programs: clean logs. If you have Stata read the commands first and run them all at the same time from memory, then your logs will only contain the output of those commands, not the commands themselves. That will keep them small and readable.

The drawback is that programs are sticky: Stata memorizes them until it is either explicitly told to forget them, or is shut down. In the same instance of Stata you might possibly run two very different do-files that call two different programs declared at different times in the past under the same name. If in this instance of Stata you only read in one of them, it will be executed twice: once in its proper context, and another time in a totally wrong one. Stata will think that that's what you mean to do.

The capture prog drop myProgram line might help a bit. Obviously you can also capture prog drop _all, with the caveat that this might drop programs that you don't want dropped. Use a blend of the two variants as circumstances warrant. Stata can live with either. The best thing to do, of course, is to use very descriptive names for programs. They usually have the side effect of being specific, which will avoid such ambiguity in the first place.