We Got Mail (Part 2)

I thought that I would talk a bit today about which stat packages are used by the ATS Statistical Consulting Group and how we select that software.

The statistical software that we support is determined by which software is used by researchers here on the UCLA campus. Just to be clear on things, there is no central administrator or committee decides which statistics software is used on campus. Each researcher, research group or department decides on their own what software they want to use.

Let's begin with general purpose statistical packages. The big three on our campus are SAS, SPSS and Stata. SAS and SPSS are legacies from the mainframe days. When I started at UCLA the top three were BMD, SAS and SPSS. BMD has dropped by the wayside and Stata has come on strong in the last seven or eight years. In addition to research usage, many departments teach their methodology courses using one or more of these stat packages.

After the big three there is R which has a smaller but very strong following. We do not get many people coming into consulting asking for help using R. I think this is due to the fact that many of the R users are relatively advanced and do not need a lot of consulting assistance. Another reason is that our group does not have lot expertise in using R. Since we don't get many question we haven't developed the expertise needed to support R at the level it deserves.

After SAS, SPSS, Stata and R there are a number of statistical packages that have a small number of users. The numbers are too small for us to invest the time and effort needed to support the software. These packages include JMP, StatView, Statistica and Datadesk. If I had written this a week ago, I would have said that there aren't any Minitab users on campus but one walked in last Thursday with a question. Fortunately, it was a more general statistical question and not something specific to Minitab. We don't have Minitab and we don't know how use it.

As for JMP, I do have a copy and have played with it some. It does some things very nicely, in fact, it does some things easily that are difficult in other packages. My opinion, based primarily on total lack of esperience, is that it might not be the best package with which to manage and analyze large research databases. If JMP grows in popularity on campus and achieves a significant number of users we would support it along with the other stat packages.

We also occasionally get a MATLAB user coming in with a question. Most of these users seem to be writing their own data analysis programs and so we can only provide limited help. I tend to lump MATLAB, Maple and Mathematica together as programming environments as opposed to traditional data analysis programs which is not to say that you can't do data analysis with them.

Then there are the special purpose statistical software packages including Mplus, EQS, LISREL, HLM, MLwiN, SUDAAN, LIMDEP, WinBUGS, LEM and LatentGOLD. We try to know a little bit about each of these, where the strengths and weaknesses lie. Different consultants in our group have differing levels of skills in these programs so clients may have to wait a bit to talk to someone with more specialized knowledge.

I need to mention one program that is invaluable to us, StatTransfer. In our line of work we could not function without this program. It allows us to move data seamlessly from stat package to another. I believe that DBMS/Copy functions in a similar manner.

Finally, I need to talk a bit about the cost of statistical software. Its expensive, even at the university discount rate. Some people think that because we are with UCLA that either we get everything for free or that we have an unlimited budget for software. Not so. Well, not completely so. In the interest of full disclosure: We do get one courtesy copy of Stata and a number of licenses for SPSS in some years. For the big three and several of the specialized packages we need licenses for all our consultants and for our lab machines. For other stat software we may only have a single copy, such as, JMP. Or, even no copies, as in the case of Minitab. Every time someone suggestion a new statistics program we need to determine if our budget can accommodate it. There is a lot of interesting statistics software that we cannot justify purchasing.

Well, that's the story on what we use in our Stat Consulting Group. I am working on a new blog entry tentatively titled, "What's the Best Stat Package." Look for it in a week or so.

pbe

We Got Mail (Part 1)

Here are two emails that we have received with comments on various statistical packages. The first email is only a few days old, while the second one goes back to last April. Part 1 contains just the emails themselves. In Part 2 we will post our comments. It will take several days to write Part 2 so that we can think of clever things to say.

James Peluso of Nassau Community College writes:

I'm fortunate to have the following packages on my home laptop:

Stata 9
JMP 6
SAS 9 (from my primary job)
Minitab 14
Maple 9

I'm like a kid in a candy store. I bought STATA thanks to your analysis, and I'm very happy with it. So I thank you for that.

I'd like to mention a terrific capability of JMP (I've used JMP for over 10 years). If you call for histograms for all of the categorical variables at the same time, you'll get a pop-up window with all of the histograms (from left to right).

Now if you click on a bar on one of the histograms, the corresponding values in the histogram bars in all of the other variables get highlighted.

JMP calls this whole process "dynamic linking"... I've used this feature countless times. It allows an analyst to quickly SEE relationships between variables.

Additionally, if you then go back to the original dataset, the corresponding records will be highlighted. This will allow the user the option to quickly create a separate dataset, which only has those records.


I think that the graphics in SAS are much better than before, thanks to ODS. But you are correct: STATA graphics are terrific.

I've just started using Minitab in my Applied Statistics course... I really like it.

I wonder if you will be analyzing Minitab in a future update to your great article?

And in response to our old podcasts Bob Solimeno of International Paper send in this email:

I just recently got an ipod and found your podcast! I enjoyed listening to the 7 podcasts published, and understand you (collectively) teach statistics with the software discussed. However, as a scientist in the corporate world many of us use Minitab or even MATLAB which gives us much more than a dedicated statistics package typically does.

Would these be possibilities for future podcasts? I would be very keen to hear your reviews of the statistics capabilities of these in contrast to MPlus, SPSS, SAS, and Stata. I realize that as educators you need to focus on a few packages and that teaching the software supplants, to some degree, the statistics curriculum. So I'll understand if my requests asks for coverage that is too broad for your podcasts.

A Special Anova

I had a client who came in with an anova question in Stata. She wanted to run a model with one between-subject factor with six levels and three within-subject factors, each with two levels. The model has seven explicit error terms, each with 30 degrees of freedom, not including the residual error which also has 30 df. The reason that there are so many error terms is that each within-subject factor has its own error term as do each of the interaction combinations of the within-subjects factors. In all, the model used 257 degrees of freedom along with another 30 df for the residual error.

We started with regular Intercooled Stata and immediately got a "too many variables or values (matsize too small error)." I set the matsize to 800, which is the maximum for Intercooled Stata and ran it again. This time it just said, "too many variables or values." Nothing about the matsize. I read in the manual that the limit for anova was eight variables in a single term. This model came close but didn't exceed that limit. Nothing we tried could get it to run.

I gave up on Stata for the moment and flipped the data into SAS using StatTransfer. Using proc glm, it ran perfectly the first time (this doesn't happen for me very often). The client was not familiar with SAS and really wanted output in Stata. I thought maybe it would run using Stata/SE. The SE in Stata/SE stands for special edition (I think) and allows a matsize up to 11,000. I got on one of the computers that had SE, set the memory to 100m and matsize to 1200 (a value I thought would be way too big), and ran it again. It worked fine, producing all the F-tests, the conservative p-values, the covariance matrix, everything.

So, what was going on? Why wouldn't it run in Intercooled Stata? A little bit of investigation revealed the answer. When I manually code a design matrix for anova, I use as many columns as their are degrees of freedom. However, Stata uses an over parameterized design matrix, it uses a many columns as there are parameters. Consider one of the with-subjects effect B*C and its error term, B*C*blocks nested in A (in Stata written as B*C*blocks|A). I would give one df for B*C and 30 df for B*C*blocks|A. Stata with its over parameterized model allocates 4 columns for B*C and 196 for the error term. In total the design needed a matsize of 1184 in order to run. So I really wasn't that far off with my wild guess of 1200.

This situation shows how quickly the matsize can grow for these mixed-effects models.

pbe

What's in a name?

As you can tell from our web pages we are the Statistical Consulting Group of UCLA Academics Technology Services. Some of the parts of our name are obvious, such as, UCLA. But the other parts may need some explaining.

"Statistical Consulting Group" -- as this part of our name suggests, we provide statistical consulting for UCLA faculty, staff and students. We provide services to campus members engaged in research. We do not provide direct support for students enrolled in statistics or other methods classes, that is, we do not assist students with homework or course projects. On the other hand, we do not charge campus researchers for any of our services, that's right, its totally free.

We have three major modes of providing our consulting services. First, we have walk-in consulting sixteen hours per week. People show up during consulting hours and receive assistance on a first come/first serve basis. On an average day we will see about a dozen clients. Our second mode is email consulting. Campus clients send in questions via email and if we can understand the question, we email a reply within one working day. If we don't understand what is being asked we can reply asking for further clarification or to ask the client to visit us during our walk-in consulting hours. When clients visit us, we ask that they bring their data with them so that we can work on their data with them. Our final mode is our web pages. We try to put up pages that cover many of the common questions and/or problems that our clients encounter. Our web services are very successful, generating nearly one million hits per month, split almost equally between campus and non-campus URLs.

"Academic Technology Services (ATS)" -- this part of our name requires a little explanation. The former name for ATS was the Office of Academic Computing (OAC). It was the campus computing center. I started at UCLA back in the mainframe days. At that time OAC was running an IBM 360 model 91 which later was swapped for an IBM 3033 and finally an IBM 3090. Just about all the statistical computing on campus was done on the central mainframe. The big stat packages in those days were BMD, SAS, SPSS. Since all this software was run centrally it made sense to have centralized statistical consultants. In fact, the vast majority of the "stat" questions concerned the JCL (Job Control Language) needed to statistics software and use tapes and disks.

As the computing environment on campus changed from centralized mainframe systems to distributed departmental and individual systems, it became clear that the name Office of Academic Computing was no longer a good fit for what the organization did. The name change to Academic Technology Services better reflects the breath of services that are currently provided. One of these services is statistical consulting. So even as computing has become decentralized, statistical consulting has remained an important component of a centralized unit, providing service to the entire campus community.
pbe

Citing the End

Most people know that we support numerous statistical software packages. But did you know that we also have a variety of other research-related programs? For example, we have three packages that help manage references: EndNote, ProCite and Reference Manager. These programs (and others like them) help you create the bibliography for your publication or dissertation. Each of them have hundreds of journal styles that you can select so that your references are in the proper format. Using a program like this may save you a headache, so UCLA researchers are welcome to stop by our walk-in consulting and try these out.

crw

Power Struggle

I got an email from Kevin Cummins (UCSD Medical School), who was playing with my powerlog program. powerlog is supposed to compute the necessary sample size to achieve a given power in logistic regression with a continuous predictor. You tell the program the proportion of one's at the mean of a continuous predictor (p1) and the proportion of one's at the mean plus one standard deviation (p2). I basically translated a SAS macro program by Michael Friendly (York University) who, in turn, referenced two formulas from Agresti's, An Introduction to Categorical Data Analysis (pg 131).

Kevin's email pointed out that my program produced some strange sample sizes. For some values of p1 and p2 the program was generating negative sample sizes and there were also times when sample size increased when the effect size increased. This, of course, is not a behavior you would expect.

The problem with the negative sample sizes was easy to correct once I spotted the error. There was a calculation that involved exp(-`lambda'^2) (the funny single quotes are Stata's way of indicating a macro variable). The way it supposed to work was that lambda was squared, then made negative and then exponentiated. The way I wrote it, lambda was first made negative, then squared and finally expopnentiated. An extra set of parentheses solved the problem.

The second problem was much more difficult to diagnose. I verified that the SAS macro program had the same problem of increasing sample size with increasing effect size under certain conditions. The algorithm used was from Hsieh (1989). So I did what I should have done in the first place, get a copy of the article from the library. I verified that Agresti, the SAS macro program and my Stata ado implemented the algorithm as it was written. Then, I did some online searching and found a much later Hsieh et al (1998) article which compared several different methods for estimating sample size for logistic regression. In the conclusion section of the article there was the statement that the algorithm should not be used for odds ratios greater than 3 or less than 1/3. With large odds ratios you can get some unbelievable sample size estimates. For example, with an odds ratio of 81 the program produces a suggested sample size of 1.7e+11.

So, right now I'm trying to decide whether to just restrict the program to odds ratios between 1/3 and 3 or to just sit on the program for a bit and search for a better algorithm that I can implement.

There will be a lot more items on power analysis coming up in the near future because we will be putting on a series of presentations on this topic in the Spring. We have an outside speaker (non-UCLA), Jason Cole (Consulting Measurement Group), who volunteered to do presentations on intermediate and advanced issues in power analysis. We will kick things off with an introduction to power analysis by our own Christine Wells. We will announce the dates and times on our ATS Stat web page.

pbe

Hey, where's the blog???

Okay, okay -- we've been really swamped in consulting. So here's a short item from yesterday to get things rolling. A student from one of the campus administrative offices comes in because she needs do a frequency distribution for items on a campus questionnaire. The catch is, they want the frequency distributions of the items to include zero frequencies, that is, when a response to one of the items is "not very often" but no one selects that response they want the table to show a zero.

This is actually fairly tricky because if no one selects that response the program doesn't "know" that the response even exists. The student's data were in Stata so I did a quick search over the internet using Stata's findit command and found a nifty program by Nick Cox called tabcount. Here's an example of how the command looks.

tabcount item1, v1(1/5)

In this example, item1 is the variable name and v1(1/5) indicates the range of possible response values. This command would have to be repeated 30 times for 30 items each with different ranges of response values.

The tabcount program would work fine for the student, but I though that I could write one that would be easier for the student to use. So, I created tablabel which uses the value labels for each of the items to define the range of values. Here is what the command looked like for the student's data.

tablabel item1-item30

And here's the output from one of the items.

Variable: item4 -- ethnicity

Value   Freq   Label
  1     24   hispanic
  2    211   asian
  3      0   african-amer
  4    143   white

Missing observations: 1

If you would like to try tablabel for yourself just type,

findit tablabel

in the Stata command line and follow the installation instructions. Remember, the variables need to have value labels already defined.

pbe