The program that presents this little problem cleans up after itself. Every week, if it's Friday, it deletes whatever "sub", "subs", "snaps" or what have you files it finds that are older than n weeks. So, eventually, after the change above, all the "subs" files will disappear. There will only be "snap" files left in their place, and the ambiguity between "sub" and "subs" will no longer be an issue.

In the meantime, though, I need a piece of code that lists file names that start with "sub" but not with "subs", which is easy enough. But it needs to work even after the last "subs" file will have left the disk. I also want it to make use of things already declared elsewhere. I basically want to be free to forget I ever wrote it long after it outlived its usefulness but, should it turn up to be needed again for some reason, I want it to pick up where it left off.  So here's how the - (minus) operator of the extended macro function list can handle that:

Right now, in the directory in question, there are four strings that file names can start with: "rat", "sub", "subs" and "snap". The first two are file names I don't want to change. Every week there will be a new file starting with either string in that directory. The last two, like I mentioned, I have more control over. The set of file names that start with "snap" is empty now but will fill up over time, and the set of file names that start with "subs" is not empty yet, but it's headed that way. I don't care about the "rat" files for this particular purpose, but whatever is irrelevant should not have to be explicitly excluded. Here's the code:

local from=strlower("`c(pwd)'")
local whichfiles "rat sub subs snap"
foreach name in `whichfiles' {
    local `name'list: dir "`from'" files "`name'*"
//  local `name'count: list sizeof `name'list
//  di "`name' count is ``name'count'"
}
local sub sub                        // (1)
local dropthese: list whichfiles-sub // (2)
di "`dropthese'"
foreach name in `dropthese' {
//  di "`name'"
    local sublist: list sublist-`name'list
//  local subcount: list sizeof sublist
//  di "now sub count is `subcount'"
}

This assumes that your directory of interest is c(pwd) -- type creturn list in Stata if this makes no sense. Notice that you have to declare the string "sub" as a local in order for the list operator minus to work -- see lines commented (1) and (2).

The beauty of the minus operator is that it acts the way you would expect it to from set theory. It will leave `sublist' unchanged if:

a) ``name'list' is empty (as in the file names that start with "subs" after the last such file will have been deleted) or

b) ``name'list' is not a proper subset (as in the file names that start with "rat", right now) or

c) both (the case of the file names that start with "snap" right now; remember I don't yet have any such files).

So I don't need to declare a `whichfiles' list with strictly the file names of interest now -- "sub" and "subs". The minus operator will gracefully handle everything. If I happen to have saved a more expansive `whichfiles' list in some other spot, for some other reason, then I can just use it from there. The lines I commented out were useful for testing, but will be a nuisance when this is live. I could have equally well enclosed the whole thing in quietly {}. Come to think of it, that would have been more elegant.

Suppose you have a file of 200,000 newspaper subscribers, half of them to The New York Times, the other to the Washington Post, and they are identified as such by a string variable named "paper" with two values -- "The New York Times" showing up 100,000 times and "Washington Post" another 100,000 times.

If "paper" were instead an integer variable with value labels "The New York Times" for 1 and "Washington Post" for 2, it would be a lot more practical. Labels need only be remembered once and your computer will have an easier time reading 1 and 2 than it would parsing the corresponding strings "The New York Times" and "Washington Post" in every single one of the 200,000 observations. The fix is simple:

encode paper, gen(x)
drop paper
rename x paper

There are some options available with encode, you can see them in the Stata help, but this will do the job: "paper" now has values 1 and 2 for the two papers. You can now save, replace and all is well: the dataset is smaller, it loads more quickly, and if you want to look at it with the data browser you will still see "Washington Post" instead of 2 on screen -- it simply shows up in blue, not red.

Now suppose you have fifty such data sets, with subscribers spanning some 150 different newspapers. And suppose you have a master list of newspapers that has 150 observations in it, where "paper" is in its original, string type. Now you have a problem. Encoding "paper" here will give you an integer with values between 1 and 150 and with value labels guaranteed to not match those in your data sets. In a data file this small, you may want to leave "paper" in string type.

There's another problem. Suppose you need to append together two data sets -- the Times/Post one in the example above and another, that combines subscribers to Baltimore Sun and Chicago Tribune. If "paper" is encoded in both sets, it has the same range of values -- 1 and 2. In the first set 1 means "The New York Times" and in the second "Baltimore Sun". Not a good thing, that.

You could recast "paper" as string, append the two datasets, then encode it again. Now it has values ranging from 1 to 4, labeled from "Baltimore Sun" to "Washington Post" in alphabetical order. But if you do this a few times with different data sets, you won't be able to keep your values and labels straight. This is just so you're aware of the problem. There are ways around it, and in all fairness sometimes the memory savings from encoding string variables aren't worth the hassle. The growing availability of 64-bit computing along with the falling cost of RAM work in your favor too.

Of course now you have the same variable in different types across datasets. What do you do if you need to operate on it inside a do-file loop that picks it out of datasets regardless of its type? One way to avoid having to remember which type "paper" is in for any given dataset is to use the extended macro function type. It works like this:

local paper_type: type paper
if regexm("`paper_type'","str") {
* perform operations on paper as if it were of type string
}
else {
* perform operations on paper as if it were of type numeric
}


Notice the regexm() expression. Stata does regular expression matching. That's a matter for another post.

Say you start out with a simple idea, like running an OLS regression. And say your dependent variable and some of the regressors are in one file, and the rest of the regressors are in another.

It's a good idea to keep your do-files short and sensible and to have them follow the general logic of the project. In this case you might split the work between two separate do-files: one collects the data needed into one joint dataset, the other runs the actual regression. Appropriate file names and judicious comments make it easy to quickly see what each do-file does and how, and all is well.

Then complications come up. The client remembers there's a third file with some regressors of potential interest, but it's in an exotic format and it has a bunch of stuff in it you don't need; or the client wants a summary report that's elaborate enough to warrant a third do-file.

Clients like it when you can accommodate gracefully their changing requirements. One easy way to do this is to keep a master do-file that points Stata to the project code directory and has it run whatever do-files it finds there without explicitly enumerating them. The extended macro function dir can do that. If you add a third project do-file, dir will find it and run it along with the others. Combine two do-files in one and dir will roll with that too.

For dir to work properly you need to help it read your files in the right succession. One way to do that is that in addition to naming these target do-files something suggestive -- like "merge first and second source file.do" -- you also pre-pend their order in the queue. If the file in this example is first in the process, simply call it "step 1 merge first and second source file.do" and number the remaining do-files in a similar fashion. That's helpful regardless of your usage of dir -- just give yourself two weeks away from the project and try to remember what goes where.

A dir-enabled master do-file will look something like this:

#delimit ;
global my_project_root=strlower("C:/My Stata Projects/This Project/Do files/");
local run_these: dir "${my_project_root}" file *;

foreach dofile in `run_these' {
do "${my_project_root}`dofile'";

/* or, if this won't work, try
do `"${my_project_root}`"`dofile'"'"';
*/

};


Notice a few things: first, I do this strlower() thing, because Stata's macro extended function dir likes its file paths in all lower case; second, I use UNIX-style forward slashes to delimit nodes in the directory path; third, when your directory and file names include spaces, you have to make Stata use adorned strings. OK, the last two items may require some additional explanations.

On using forward slashes: I think this is a good habit to pick up. Windows doesn't care either way, and forward slashes eliminate the possibility of misleading Stata into thinking that a directory delimiter is an escape order. That alone is a good reason to just never use backslashes in file paths.

Adornments are Stata speak for preserved quotation marks. Adorned strings obviously require compound quotes. How you compound them takes a little trial and error. That is a bit of a pain, but there's no way around it if you insist on spaces in file names.

If, for example, you have two do-files, one called "foo 0.do" and the other "bar 1.do", adornments help the dir function correctly establish that the local macro `run_these' is a list of two items (as in ""foo 0.do" "bar 1.do"") and not a string of four words (as in "foo 0.do bar 1.do"). Adornments and compound quotes make it possible to use spaces in file names. Spaces help legibility. It's up to you whether that benefit is worth the extra cost of more complicated syntax.

But that was a digression. The point of this post was the extended macro function dir. This function allows you to have Stata read whatever do-files you put in a directory, so that if your project scope changes over time your list of do-files is free to change along with it. I think that's a neat option to have.