I was reminded of this when I tried to run freduse today. Actually, the problem that reminded me of it -- a "not found" error message thrown when the command invoked the Mata function _fredifinparse() -- didn't go away, but "adoupdate, update" is still a good thing to do. What did fix _fredifinparse() is described here.

My Stata code between 2000 and 2006 consisted exclusively of do-files that put to work either standard Stata commands or user-written commands from the SSC. There was not a single program definition anywhere and things worked alright. These do-files were pretty elaborate and their functionality overlapped a fair bit, but that was never that much of an inconvenience.

Then in early 2007, during my brief tenure at RTI Health Solutions, that way of working showed its limitations when I tried to program in plain Stata matrices something that was normally being done in GAUSS. It had to do with the design of factorial experiments and my project ended in an instructive kind of failure, because it got me started on using Stata programs. I still like those things. I can define them once and then nest and have them call each other every which way to my heart's content. They take arguments, return values, and generally they make you feel like a real programmer.

Then in 2008 I had my introduction to C++, and the instructions were clear: break down a problem in small morsels; use as many functions as you need; if a function definition fills up a screen, it's way too big, so break it down further; encapsulation is a good thing. Then came header files, namespaces, classes, templates, the works. It was an extreme kind of validation of the way I had started to do business, and my enthusiasm for modular code only grew from there.

Then, about a year ago, I started running into problems. Component programs can be debugged individually, sure, and you only need to fix them once, in one place, which is great. In fact, if they're small and simple enough, you don't even need to do that; they just work. But with complicated projects you're going to have so many interlocked small and simple programs that it will just be too hard to keep tabs on which programs call which, where, and why. It's also pretty expensive to write them in such a way that they can talk with not just one other program, but are truly universal within the context of the given problem.

So I'm not sure anymore that I would recommend my way of writing Stata code to everybody. It still has its uses, but I can see a growing number of circumstances where it's simply not worth the trouble.

This post is the result of the conversation I started in the comment thread with Gabriel Rossman. I will attempt to use Mata for string processing within a suitably large text file, as opposed to just a blob of text you can call as a local.

Step 1: google "very large text file". This took me to a magical place where the 1980's are preserved in perpetuity. I went through the categories, and picked this one. At exactly 12,133 lines, it should do nicely.

Step 2: get the Mata book -- because I still run Stata 10 at home, so no pdf documentation yet.

Step 3: muck around. Eventually I came up with this thing:

mata
real scalar checkmatch(string scalar theFile, string scalar thePattern)
{
   real scalar n,i,check
   string matrix A
   A=cat(theFile)
   n=rows(A)
   check=0
   for(i=1; i<=n; i++) {
      if(strmatch(A[i,1],thePattern)) {
         check=1
         return(check)
      }
   }
   return(check)
}
end

This is a Mata function that returns 1 if a string pattern is found anywhere in a given text file, and 0 otherwise. It makes use of Mata's built-in cat() function, which reads an ASCII file of n lines into a column vector of n string elements, one for each line in the original file. I want checkmatch() to exit with 1 as soon as it first finds the string pattern it's looking for. I'm guessing that the first return(check), inside the if clause, does it, but I'm not sure.

With a text file this big, the 0 case might be the harder one to test, but if you're fishing for patterns you're unlikely to find in an English-language document no matter how big, a Hungarian word is a pretty good bet. So, this is the output:

. mata: checkmatch(`"dostech.pro"',`"*BIOS*"')
  1
. mata: checkmatch(`"dostech.pro"',`"*Kolozsvar*"')
  0

Now for a real illustration of Mata's string and file processing capabilities, see here.

A search on how to best fill them had to start with the Statalist archive. I quickly found this simple solution by Scott Merryman, which solved the core of the problem. But then things got complicated. First, I had several variables of interest and I had no reason to expect that they all would have a linear time trend. Second, some of my variables -- such as measured groundwater salinity -- showed pretty unbelievable outliers.

So I needed some kind of wrapper that would accommodate different variables, trends of different orders, and dummy variables to account for any differences between counties or groups of counties. And I needed a quick and easy way to deal with the outliers. The wrapper is below:

// #### higher-order predictions of anything against time.
// takes three arguments:
// 1. order, numeric= 1,2,3,4,5,etc. for linear, squared,
//    cubed, etc. trend
// 2. lhs, string, the name of the left-hand side variable:
//    salinity, usage, population, etc.
// 3. dummy, string -- name of the group identifier on the
//    right-hand side, e.g., county fips code.
capture prog drop getValueHat
program getValueHat

args order lhs dummy

levelsof `dummy', local(countem)
local checkit: list sizeof countem
// if `dummy' is not degenerated to one value, then use xi: regress
if `checkit'>1 {
   local regressors i.`dummy'*year1
   forvalues i=1/`order' {
      gen year`i'=year^`i'
      if `i'>1 {
         local regressors `regressors' i.`dummy'*year`i'
      }
   }
   xi: regress `lhs' `regressors'
}
// otherwise, just regress
else {
   local regressors year1
   forvalues i=1/`order' {
      gen year`i'=year^`i'
      if `i'>1 {
         local regressors `regressors' year`i'
      }
   }
   regress `lhs' `regressors'
}
predict `lhs'_hat
replace `lhs'_hat=max(0,`lhs'_hat)
if `checkit'>1 {
   drop _I*
}
forvalues i=1/`order' {
   drop year`i'
}

end

Now I can use the same code for different variables with different functional forms, which is nice. For example, if I want to fill gaps in salinity at the county level with a quadratic trend, this program call will get me the right salinity_hat:

getValueHat 2 salinity county

Dealing with the outliers was a far simpler matter: Nick Cox's winsor command -- findit winsor if you don't have it installed. The complete solution to my problem of gaps and outliers is:

winsor salinity, h(3) gen(x)
getValueHat 2 x county
rename x_hat salinity_hat
drop x

If, like me, you've never Winsorized before, here's the Wikipedia entry on the procedure.

He suggests, in Chapter 5.2, a nice do-file method to validate your data: you use pairs of list and assert. For example, suppose you know that a variable v should have no missing values. If it indeed does not, then assert !missing(v) should run without error. If it does, you want to know where they are: list if missing(v). Reversing the order of these two lines in a do-file will cause Stata to exit with an error, which alerts you that there are problems with your data, but not before it shows you where the problems are:

sysuse auto, clear
list if missing(make)
assert !missing(make)
list if missing(rep78)
assert !missing(rep78)

This is neat because you can always edit this do-file with a new pair of list/assert lines as needed. But, as the author mentions, sometimes a summarize is plenty helpful too, especially if you remember that it accepts a list of variables as an argument. You could do this for example:

sysuse auto, clear
sum make rep78

Wait. That didn't work too well, because summarize tells you nothing about make: it treats string variables as missing. You would know that that was indeed the case if before sum you would have requested describe.

So, I guess, before you do any kind of data validation, describe is a good first step; you might also like codebook; I don't. I find it too wordy. But it does do the job of giving you information about the whole data set.

One alternative to wordy output when you have a specific question regarding more than one variable is to use little custom programs for data checks. On such example is countIfMissing, shown in my previous post. Another, inspired by Christopher Baum's use of summarize might be sumIfNumeric:

capture prog drop sumIfNumeric
program sumIfNumeric

unab fullset: _all
local numset
foreach varble in `fullset' {
   capture confirm numeric variable `varble'
   if _rc==0 {
      local numset `numset' `varble'
   }
}
local check: list sizeof numset
if `check'>0 {
   sum `numset'
}
else {
   di "No numeric variables found in this dataset."
}

end

This might be useful when your data set comes with a bunch of variables -- some numeric, some string. Though describe will tell them apart easily enough, you may not care to list them explicitly. The usage is straightforward:

sysuse auto, clear
sumIfNumeric

Here is one suggestion:

// countIfMissing: display the total count of observations, then
// any counts of missing observations for each variable in a list.
capture prog drop countIfMissing
program countIfMissing

version 11
syntax varlist

quietly count
local count=r(N)

// now make things align nicely
local sum=`count'
local tens=1
while `sum'/10>1 {
   local sum=`sum'/10
   local tens=`tens'+1
}
local width=`tens'+int(`tens'/3)
local varct: list sizeof varlist

di ""
di "Observations:"
di %`width'.0fc `count'
di ""
di "Missing:"
foreach varble in `varlist' {
   qui count if missing(`varble')
   local ct=r(N)
   local pct: di %4.2fc 100*`ct'/`count'
   if `pct'>0 {
      di %`width'.0fc `ct' " `varble' (`pct'%)"
   }
   else {
      local varct=`varct'-1
   }
}
if `varct'==0 {
   local offset=`width'+2
   di _column(`offset') "none of `varlist'"
}
di ""

end

For an example of usage, you can try this:

sysuse auto
local myvars "make price foreign"
countIfMissing `myvars'
countIfMissing m*       // (1)
countIfMissing _all     // (2)

As you can see in (1) and (2), the usual varlist conveniences apply here.

There was one line that I had to change, where Vim is ordered to clean up after itself. Every time you runDoLines() you produce a bunch of .tmp.do files in your %temp% directory, which Vim knows as $TEMP. Though this

au VimLeave * exe "!del -y " temp

should have deleted them, on my computer it did not. This one did:

au VimLeave * silent exe '!del /Q "'.$TEMP.'\*.tmp.do"'

Another thing I tinkered with was the way Vim handles these backup files that end in a tilde. You can disable them entirely, by adding

set nobackup nowritebackup

to _vimrc as explained here. You can also have Vim keep them somewhere you can delete them explicitly, in bulk, as explained here. Finally, you can use a combination of both: you save backup files while Vim is working, but then you make it delete them all when it starts next time. That is explained here. I went with it, so after I created a temp folder in $VIMRUNTIME, I added this to my _vimrc file:

" Keeps backups in $VIMRUNTIME\temp folder, cleans them up
silent execute '!mkdir "'.$VIMRUNTIME.'\temp"'
silent execute '!del /Q "'.$VIMRUNTIME.'\temp\*~"'
set backupdir=$VIMRUNTIME\\temp\\
set directory=$VIMRUNTIME\\temp\\

In the process of figuring this out, I also discovered that there's a quick way to clean up %temp% thoroughly, as shown here. Good to know. One of these days I might add that to _vimrc too. Now I'm on a roll.

I wrote a do-file, called work.do, that just uses a dataset, nothing more. That data set's name is handled by a local macro, defined in a separate do-file, called locals.do, which work.do includes. Then I wrote locals_program.do which does the same job via a program, which then work_program.do calls by name. Finally, I wrote a profiler.do file that called all four files a few times, and measured the time they all took doing their thing. According to profiler.do, an include is usually faster than a program. Below is the code:

// work.do starts here:
include locals.do
use "`my_file'"

// locals.do starts here
local file_path "C:/work/romanian papers circulation figures/data/"
local file_name "file_combined.dta"
local my_file   "`file_path'`file_name'"

// this is locals_program.do
capture prog drop defineMyLocals
program defineMyLocals, rclass

local file_path "C:/work/romanian papers circulation figures/data/"
local file_name "file_combined.dta"
local my_file   "`file_path'`file_name'"

local things "file_path file_name my_file"
foreach thing in `things' {
   return local `thing' ``thing''
}

end

// this is work_program.do
defineMyLocals
local my_file `r(my_file)'
use "`my_file'"

// and finally, this is profiler.do

set more off

cd "c:/work/programming/putterin/profiler"

// SECTION 1: OVERVIEW
/*
   1. What's going on:
   "program" vs. "include" profiler. this do-file defines
   two programs that will measure the performance difference
   in defining locals separately using two alternate ways
   (1) locals are defined in a do-file called with include
   (2) they are defined in a program called by name

   The exercise is to "use" a file. This file is called via
   a handle defined as a local macro. That definition can be
   either in a do-file included (1) or in a program called
   by name (2).

   2. Programs defined here and their dependencies:
   runProfile1
      defineMyLocals // defined in locals_program.do
   runProfile2
*/

// SECTION 2: GLOBALS

// SECTION 3: PROGRAM DEFINITIONS

// ### programs defined elsewhere and called via "run"
// ### locals_program.do defines the program named
// ### defineMyLocals, which returns a file handle
// ### as an r() local.
run locals_program.do

// ### work_program.do uses a file whose handle
// ### comes from calling defineMyLocals and
// ### retrieving an r() local.
capture prog drop runProfile1
program runProfile1

args counter

local time_start=tc("`c(current_date)'" "`c(current_time)'")
forvalues i=1/`counter' {
	run work_program.do
}
drop _all
local time_end=tc("`c(current_date)'" "`c(current_time)'")
di ""
di "Profile 1 (program), `counter' reps"
di "Time elapsed (ms): "`time_end'-`time_start'

end

// ### work.do includes locals.do, which
// ### defines a file handle as a local.
// ### work.do uses that file by calling
// ### that local.
capture prog drop runProfile2
program runProfile2

args counter

local time_start=tc("`c(current_date)'" "`c(current_time)'")
forvalues i=1/`counter' {
	run work.do
}
drop _all
local time_end=tc("`c(current_date)'" "`c(current_time)'")
di ""
di "Profile 2 (include), `counter' reps"
di "Time elapsed (ms): "`time_end'-`time_start'

end

// SECTION 4: PROGRAM CALLS

local cycles "100 200 500 1000 1500"
foreach cycle in `cycles' {
	forvalues i=1/2 {
   		runProfile`i' `cycle'
	}
}

And that's it. Put all five files into the same directory, make the profiler cd to it, change the path and name locals to your own data set, and see what you get on your machine. Below is my output:

Profile 1 (program), 100 reps
Time elapsed (ms): 2000

Profile 2 (include), 100 reps
Time elapsed (ms): 2000

Profile 1 (program), 200 reps
Time elapsed (ms): 4000

Profile 2 (include), 200 reps
Time elapsed (ms): 3000

Profile 1 (program), 500 reps
Time elapsed (ms): 9000

Profile 2 (include), 500 reps
Time elapsed (ms): 8000

Profile 1 (program), 1000 reps
Time elapsed (ms): 18000

Profile 2 (include), 1000 reps
Time elapsed (ms): 17000

Profile 1 (program), 1500 reps
Time elapsed (ms): 26000

Profile 2 (include), 1500 reps
Time elapsed (ms): 26000

Vim does not get confused by compound quotes and it comes with Stata syntax highlighting out of the box. Integrating it with Stata is not hard at all, either. There are some general directions by Nick Cox here, but on my Windows XP machine I just had to do two things.

First, I had to edit the PERSONAL/e/editors.ado file to make Vim replace Notepad++ entry in the Editors... sub-menu of the User menu:

program define editors
	version 10
	window menu append submenu "stUser" "Editors"
	window menu append item "Editors" "Vim" "winexec gvim.exe"
end

Next, I wrote a PERSONAL/v/vim.ado file like so:

program vim
	version 10
	syntax anything
	winexec gvim.exe `anything'
end

This is based on Nick Cox's recipe, slightly altered so you can launch Vim from the Stata command line to edit an existing do-file that say has spaces in it, as in vim "my dofile.do". There's a little more on this "syntax anything" solution in my previous Stata post.

One thing I did give up -- for now -- is the ability to launch a Stata instance from within Vim after I'm done editing a do-file, either for running the entire do-file or some select lines, the way I used to be able to do in Notepad++. There's got to be a way for that, though. If you know it, please drop me a line.

That said, I'd never knock Notepad++. It is still a fine text editor that has served me well. The people at Mather Economics, who adopted it a while back at my instigation, are sticking with it -- and I recommend it to anybody who does not insist on a modal editor.

syntax namelist
local input_file `namelist'

That, however, choked on a Windows-style file path, because the colon in C:\ is an illegal name. Trial and error (and the [I] book) led me to syntax anything. Here's my scratchpad:

capture prog drop myHello
program myHello

syntax namelist
local names `namelist'
di "Hello `names'"

end

myHello Eenie: Meenie

OK, so "illegal name" it is. Now let's see if "anything" might work better than "namelist":

capture prog drop myHello
program myHello

syntax anything
local names `anything'
di "Hello `names'"

end

myHello c:\my file path here\my file name here.txt

Success. This may look trivial, but everything is after the fact. We have a long-standing policy at the Eno River Analytics worldwide headquarters that we should write toy programs for sketching things out before we go ahead and actually break something.