Remove Spotify App Badges on Mac

See here

cpumemlog: Monitor CPU and RAM usage of a process (and its children)

Long time no see ...

Today I pushed the cpumemlog script to GitHub https://github.com/gregorgorjanc/cpumemlog. Read more about this useful utility at the GitHub site.

Read line by line of a file in R

Are you using R for data manipulation for later use with other programs, i.e., a workflow something like this:

1. read data sets from a disk,
2. modify the data, and
3. write it back to a disk.

All fine, but of data set is really big, then you will soon stumble on memory issues. If data processing is simple and you can read only chunks, say only line by line, then the following might be useful:

## File
file <- "myfile.txt"
 
## Create connection
con <- file(description=file, open="r")
 
## Hopefully you know the number of lines from some other source or
com <- paste("wc -l ", file, " | awk '{ print $1 }'", sep="")
n <- system(command=com, intern=TRUE)
 
## Loop over a file connection
for(i in 1:n) {
  tmp <- scan(file=con, nlines=1, quiet=TRUE)
  ## do something on a line of data 
}

Created by Pretty R at inside-R.org

Parse arguments of an R script

R can be used also as a scripting tool. We just need to add shebang in the first line of a file (script):

#!/usr/bin/Rscript

and then the R code should follow.

Often we want to pass arguments to such a script, which can be collected in the script by the commandArgs() function. Then we need to parse the arguments and conditional on them do something. I came with a rather general way of parsing these arguments using simply these few lines:

## Collect arguments
args <- commandArgs(TRUE)
 
## Default setting when no arguments passed
if(length(args) < 1) {
  args <- c("--help")
}
 
## Help section
if("--help" %in% args) {
  cat("
      The R Script
 
      Arguments:
      --arg1=someValue   - numeric, blah blah
      --arg2=someValue   - character, blah blah
      --arg3=someValue   - logical, blah blah
      --help              - print this text
 
      Example:
      ./test.R --arg1=1 --arg2="output.txt" --arg3=TRUE \n\n")
 
  q(save="no")
}
 
## Parse arguments (we expect the form --arg=value)
parseArgs <- function(x) strsplit(sub("^--", "", x), "=")
argsDF <- as.data.frame(do.call("rbind", parseArgs(args)))
argsL <- as.list(as.character(argsDF$V2))
names(argsL) <- argsDF$V1
 
## Arg1 default
if(is.null(args$arg1)) {
  ## do something
}
 
## Arg2 default
if(is.null(args$arg2)) {
  ## do something
}
 
## Arg3 default
if(is.null(args$arg3)) {
  ## do something
}

## ... your code here ...

Created by Pretty R at inside-R.org

It is some work, but I find it pretty neat and use it for quite a while now. I do wonder what others have come up for this task. I hope I did not miss some very general solution.

Functions dim, nrow, and ncol for shell

Ever wanted to find out quickly the number of rows and/or columns in file directly from terminal. There are many ways to skin this cat. Here is what I used for number of rows for quite a while:

wc -l filename

What about number of columns? "Easy", just combine head and awk commands:

head -n 1 filename | awk '{ print NF }'

not a big problem (there are likely better ways to do this), but is long and tedious.

I got sick of typing commands above and assembled them in three easy to use function with R-like names: nrow, ncol, and dim. Functions are simply a collection of above ideas and assume that the file is of "rectangular shape", i.e., a table, a matrix, etc.

dim()
{
  for FILE in $@; do
    NROW=$(nrow $FILE | awk '{ print $1}')
    NCOL=$(ncol $FILE | awk '{ print $1}')
    echo "$NROW $NCOL $FILE"
    unset NROW NCOL
  done
}
export -f dim

nrow ()
{
  for FILE in $@; do
    wc -l $FILE
  done
}
export -f nrow

ncol ()
{
  for FILE in $@; do
    TMP=$(head $FILE -n 1 | awk '{ print NF }')
    echo "$TMP $FILE"
    unset TMP
  done
}
export -f ncol

Add these files to your .bashrc or .profile or something similar and you can now simply type:

nrow filename

ncol filename

dim filename

A simple test:

touch file.txt

echo "line1 with four columns" >> file.txt

echo "line2 with four columns" >> file.txt

nrow file.txt

2 file.txt

ncol file.txt

4 file.txt

dim file.txt

2 4 file.txt

Recover deleted file under linux

Here is a nice summary, but the bottom line is that the Ubuntu/Debian package testdisk has a nice utility called photorec that can be used to search for deleted files and recover. This is not a GUI program, though!

AWK one-liners

See here.

Google drive drives in

I just realized that Google offers me "Google drive". The pricing options for larger disk space than 5GB are very competitive with Dropbox! I am using Dropbox for all my files - yes, it costs me quite some money when the bill comes, but if we consider this is "few" dollars per month for backup and synchronization I am more than willing to pay this amount for this!!! Google drives is now yet another option, but the desktop folder does not work on Linux so for now I will stick with Dropbox.

Compress PDF

See nice description here. In short using the command bellow on my Linux box (Windows users see PDFCreator) allowed me to reduce scanned color PDF for a factor of 4 to 5, e.g., from ~14 MB to  ~3 MB. Supposedly PDFCreator also 

gs -sDEVICE=pdfwrite -dMaxSubsetPct=100 -dPDFSETTINGS=/ebook -sOutputFile="out.pdf" -dNOPAUSE -dBATCH "in.pdf"


Update (2012-01-16): you can get a nice shell script called compressPDF.sh that saves some typing and can process multiple files, e.g., 


compressPDF.sh file.pdf

Agilent's Integrated Biology Grant Program

Fostering integrated, whole-systems approaches to biological research with two $75,000US grants for open source data-integration tool development. See more here.

Redish color of screen for evening or night work

While reading local computer magazine (Monitor) I find out about f.lux, which is program that changes the color of screen to more redish for evening or night work. They claim that this protects our eyes and sleep pattern. I like the idea. Linux support has some problems, but their is alternative that works with Linux - redshift (see here and here for installation).

Nice simple notes on running R in parallel by Geyer

Here, by Charles J. Geyer.

bugsparallel

bugsparallel is a Metrum Institute project to run BUGS (via R2WinBUGS) in parallel - McMC is an application, where parallel runs can be used very efficientlly. Here is the code for one example using bugsparallel.

Some usefull links:

• Rosenthal, Parallel computing and Monte Carlo algorithms
• Wilkinson, Parallel Bayesian computation

Sweave.sh update

A local version of Sweave.sh has been updated:

• fixed a buglet that caused weaving the file twice when --weaver option was used --> this led to a change how caching is now invoked (--cache invokes cacheSweave package, while --cache --weaver or --weaver invokes weawer package)
• quoting *APP variables to ensure that things work in case of "bad" filenames, i.e., spaces in filenames etc.
• small changes in the documentation

I will take this opportunity to show how PDFAPP and PSAPP environmental variables can be used "open" options. This is an example from the help, which shows that these two variables need to be exported in order to have any effect. I am still fiddling whith how to provide any meaningfull defaults. My defaults are 'acroread' for PDF and 'gv' for Postscript, but I am happy to set any other more meaningfull defaults if there will be enough demand!

# Create PDF via the texi2dvi (latex) tool and open
# a produced file
Sweave.sh -otld file.Rnw

# Create PDF via the texi2dvi (latex) tool and open
# a produced file with a "non-standard" viewer
Sweave.sh -otld=acrobat file.Rnw

# ... or
export PDFAPP=acrobat
Sweave.sh -otld file.Rnw

This is an example of launcing Acrobat Reader installed on MS Windows, while Sweave.sh is launched within Cygwin X-terminal:

export PDFAPP=/cygdrive/c/Program\ Files/Adobe/Reader\ 9.0/Reader/AcroRd32.exe
Sweave.sh -otld file.Rnw

ATLAS

ATLAS (link1, link2) is a Java based tool to manage genotypes. It is handy, but notoriouslly frustrating to install, since there are several different instructions about how to install it - I am not talking about different instructions at different places (say on the net), but in the distribution package. This is very very confusing. I have just spent more than half an hour to install it and I remember I had the same problem last time I tried to install it. The correct instructions are in the ATLAS.html file and this should be done:

1. Create ATLAS directory in your user home directory, say "C:\Documents and Settings\GGorjan"
2. Add the following files to this folder (Atlas.jar, atlas.ini, chr.atl, ATLASman_v1.4.pdf, ATLAS.html)
3. Create directory ATLAS_files such as "C:\Documents and Settings\GGorjan\ATLAS\ATLAS_files" and put all figures and html files in there
   

Accessing files (read and write) with dual boot (Windows & Linux)

If you have a computer with a dual boot system (you have installed Windows and Linux on the same computer and you decide which one to use at computer startup) you can easily access files in read and write mode using EXT2 IFS on the Windows side and Linux-NTFS on the Linux side. I have used both of them for quite some time without any problems.

Update: I experienced a problem with EXT2 IFS with accessing the disk partitions on a freshly installed Ubuntu 8.10. Using the mountidag tool (as recommended here) I get this message:

C:\ggorjan\Desktop>mountdiag d:
The volume has an Ext2/Ext3 file system, but the Ext2 IFS 1.11 software did not
mount it because the file system has an inode size unequal to 128 bytes (inode
size: 256 bytes).
The only way to solve it is to back up the volume's files and format the file
system: give the mkfs.ext3 utility the -I 128 switch. Finally, restore all
backed-up files.
After that, the Ext2 IFS software should be able to access the volume.

I did not see this error in the past. I did some googling and found out that new linuxes use larger inode size (whatever this means). I could reformat the linux partitions, but I found out that there are also other solutions beside EXT2 IFS - see this article for more info. Since I want the write support it comes down to use either EXT2 IFS or Ext2fsd. I was a bit afraid to use Ext2fsd, because I do not know it, but I also did no know EXT2 IFS in the past, so I guess that posts at Ubuntu forums gave me confidence. What I did:

• installed Ext2fsd
• assigned the drive letters
• reboot
• after that I was not able to create a new file on a Linux (Ext3) partition (using the MS Windows OS) --> went to Ext2fsd folder and started the Ext2 Volume Manager --> right click on a partition --> Ext2 Management --> Check out the readonly option
• reboot
• voila!
  

Version control (CVS and SVN) cleints for MS Windows

Version control is a good thing. I have some experience with CVS and SVN. Installing the clients to "talk" with the server is very simple under Linux, i.e., something like:

apt-get install cvs svn

It is quite some time since I "discovered" the TortoiseSVN, which is a client for MS Windows. It is really nice, since it integrates smoothly with the Windows Explorer (the yellow folder icon). Today, I also "discovered" that there is also TortoiseCVS. I usually do not do any development under MS Windows, but it is very handy to be able to do a change or two and commit the changes from MS Windows.

In my opinion using the open source programs (R, FireFox, ThunderBird, ...) can improve the working experience with MS Windows a lot.

Nemške ambasade gredo na Linux

Na Slo-Tech-u so objavili novico "Nemške ambasade gredo na Linux". Zanimiva novica. Menim tudi, da so nekateri komentarji zelo relevantni.

Excel 2007 for statistics?

Microsoft Excel is probably one of the most used software for performing various "calculations". I agree that spreadsheets are very usefull and for some tasks MS Excel just excels. However, MS Excel has problems when it somes to statistics. There have been quite some reports about the inaccuraccy of statistical routines. For the 2007 version, McCullough and Heiser (2008)

wrote this in the abstract

Excel 2007, like its predecessors, fails a standard set of intermediate-level accuracy tests in three areas: statistical distributions, random number generation, and estimation. Additional errors in specific Excel procedures are discussed. Microsoft’s continuing inability to correctly fix errors is discussed. No statistical procedure in Excel should be used until Microsoft documents that the procedure is correct; it is not safe to assume that Microsoft Excel’s statistical procedures give the correct answer. Persons who wish to conduct statistical analyses should use some other package.

MS Excel is not the only spredsheet software. Yalta (2008) compared MS Excel with some others and wrote this in the abstract:

We provide an assessment of the statistical distributions in Microsoft® Excel versions 97 through 2007 along with two competing spreadsheet programs, namely Gnumeric 1.7.11 and OpenOffice.org Calc 2.3.0. We find that the accuracy of various statistical functions in Excel 2007 range from unacceptably bad to acceptable but significantly inferior in comparison to alternative implementations. In particular, for the binomial, Poisson, inverse standard normal, inverse beta, inverse student’s t, and inverse F distributions, it is possible to obtain results with zero accurate digits as shown with numerical examples.

McCullough (2008) contrubuted also this:

Microsoft attempted to implement the Wichmann–Hill RNG in Excel 2003 and failed; it did not just produce numbers between zero and unity, it would also produce negative numbers. Microsoft issued a patch that allegedly fixed the problem so that the patched Excel 2003 and Excel 2007 now implement the Wichmann–Hill RNG, as least according to Microsoft. We show that whatever RNG it is that Microsoft has implemented in these versions of Excel, it is not the Wichmann–Hill RNG. Microsoft has now failed twice to implement the dozen lines of code that define the Wichmann–Hill RNG.

Additionally, the graphs made in MS Excel are not really good ones. It depends a lot on the user to make a good graph. I have seen many so called chartjunk commming from Excel. Of course the blame is on users, but software should help us make things easier and not harder! Su (2008) wrote this in the abstract

The purpose of default settings in a graphic tool is to make it easy to produce good graphics that accord with the principles of statistical graphics... If the defaults do not embody these principles, then the only way to produce good graphics is to be sufficiently familiar with the principles of statistical graphics. This paper shows that Excel graphics defaults do not embody the appropriate principles. Users who want to use Excel are advised to know the principles of good graphics well enough so that they can choose the appropriate options to override the defaults. Microsoft® should overhaul the Excel graphics engine so that its defaults embody the principles of statistical graphics and make it easy for non-experts to produce good graphs.

After all this anti-Microsoft abstracts I really, really start to wonder what we get for the money we spend for the software.

Using optimized linear algebra libraries with R

After reading blog posts by Yu-Sung and Gelman (see also comments) as well as recent additions to R on Debian page at RWiki, I decided to take a try with ATLAS on my laptop. I did not aspect much, since I use a poor IBM ThinkPad R50e with Intel(R) Celeron(R) M processor - 1400MHz. However, this laptop offers what I need when I move around, while I use a "hammer" in the office. I could not find the information about which instruction set (SSE, SSE2, ...) is appropriate for my processor. In order to figure this out and to evaluate the benefits I did:

• Launched the XUbuntu 7.10 (gutsy)
  
• Started R with: R --vanilla -q
• Used the following R script i.e. setting up the matrix and calculating its crossproduct
  

mm <- matrix(rnorm(4 * 10^6), ncol = 10^3); system.time(crossprod(mm))

• Tested packages: atlas3-base, (atlas3-sse, atlas3-sse-dev) and (atlas3-sse2, atlas3-sse2-dev), which is just a matter of installing and removing the packages via aptitude install or remove command

The results (output from system.time function - several calls) are:

• standard R installation: user ~13.4, system ~0.06, elapsed ~15.0
  
• atlas3-base: user ~4.1, system ~0.05, elapsed ~4.2
  
• atlas3-sse: user ~3.4, system ~0.03, elapsed ~3.7
  
• atlas3-ss2: user ~4.1, system ~0.03, elapsed ~4.1
  

It is clear that installing ATLAS is beneficial, but there is not much gain with specific instruction sets for my laptop. Nevertheless, I kept atlas3-sse since it was consistently showing the best performance.