cronolog and STDERR

At work we use cronolog for automatic rotation of log files for a number of processes since those processes just write to STDOUT and STDERR instead of using a proper logging library. Unfortunately, that means when running the script/program we have to redirect STDERR to STDOUT, and then pipe the results to cronolog, since cronolog reads from STDIN. The result looks something along the lines of the following:

ruby main.rb &2>1 | cronolog /logs/main.log /logs/main-%Y-%m-%d.log

The problem with this is if that errors are few and far between, as one hopes they should be, then it might be really tricky to find the errors amongst the other logging. Ideally, I thought it would be nice to have STDOUT go to one log file, and STDERR get written to a err file for the process.

After some digging into From Bash to Z Shell I found something about process substitution in the Bash shell. After a little experimentation and tweaking, I came up with the following:

ruby main.rb \
     > >(/usr/sbin/cronolog /logs/main.log /logs/main-%Y-%m-%d.log) \
     2> >(/usr/sbin/cronolog /logs/main.err /logs/main-%Y-%m-%d.err)

This allows me to use cronolog with both the STDOUT and STDERR streams. By using cronolog in the process substitution, it allows the output streams to be treated as input streams to cronolog, where as before I had to combine them into one stream and then pipe the single stream to cronolog as in the first example.

Hope this can help someone else, and save some hours of digging.

–Proctor

Error installing iconv

Today I was trying to install the iconv gem, and was getting this error

$gem install iconv
Building native extensions.  This could take a while...
ERROR:  Error installing iconv:
	ERROR: Failed to build gem native extension.

        <rvm_dir>/rubies/ruby-2.0.0-preview2/bin/ruby extconf.rb
checking for rb_enc_get() in ruby/encoding.h... yes
checking for iconv() in iconv.h... no
checking for iconv() in -liconv... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=<rvm_dir>/rubies/ruby-2.0.0-preview2/bin/ruby
	--with-iconv-dir
	--with-iconv-include
	--without-iconv-include=${iconv-dir}/include
	--with-iconv-lib
	--without-iconv-lib=${iconv-dir}/
	--enable-config-charset
	--disable-config-charset
	--with-config-charset
	--without-config-charset
	--with-iconvlib
	--without-iconvlib


Gem files will remain installed in <rvm_dir>/gems/ruby-2.0.0-preview2/gems/iconv-1.0.2 for inspection.
Results logged to <rvm_dir>/gems/ruby-2.0.0-preview2/gems/iconv-1.0.2/ext/iconv/gem_make.out

After a bit of searching, I found a number of answers suggesting that I would need to reinstall ruby via RVM, but in the following StackOverflow question
failed-to-build-iconv-gem-on-ruby-1-9-2.

In it was the solution:
gem install iconv -- --with-iconv-dir=/usr/local/Cellar/libiconv/1.13.1

After that, all was well.

–Proctor

Log File parsing with Futures in Clojure

As the follow up to my post Running Clojure shell scripts in *nix enviornments, here is how I implemented an example using futures to parse lines read in from standard in as if the input was piped from a tail and writing out the result of parsing the line to standard out.

First due to wanting to run this a script from the command line I add this a the first line of the script:

 
!/usr/bin/env lein exec

As well, I will also be wanting to use the join function from the clojure.string namespace.

 
(use '[clojure.string :only (join)])

When dealing with futures I knew I would need an agent to adapt standard out.

(def out (agent *out*))

I also wanted to separate each line by a new line so I created a function writeln. The function takes a Java Writer and calls write and flush on each line passed in to the function:

(defn writeln [^java.io.Writer w line]
  (doto w
    (.write (str line "\n"))
    .flush))

Next I have my function to analyze the line, as well as sending the result of that function to the agent via the send-off function.

(defn analyze-line [line]
  (str line "   " (join "  " (map #(join ":" %) (sort-by val > (frequencies line))))))

(defn process-line [line]
  (send-off out writeln (analyze-line line)))

The analyze-line function is just some sample code to return a string of the line and the frequencies of each character in the line passed in. The process-line function takes a line and calls send-off to the agent out for the function writeln with the results of calling the function analyze-line.

With all of these functions defined I now need to just loop continuously and process lines that are not empty, and call process-line for each line as a future.

(loop []
  (let [line (read-line)]
    (when line
      (future (process-line line)))
      (recur)))

Running Clojure shell scripts in *nix environments

I was recently trying to create a basic piece of Clojure code to play with “real-time” log file parsing by playing with futures. The longer term goal of the experiment is to be able to tail -f a log file pipe that into my Clojure log parser as input.

As I wasn’t sure exactly what I would need to be doing, I wanted an easy way to run some code quickly without having to rebuild the jars through Leiningen every time I wanted to try something, in a manner similar to the way I am thinking I will be using it if the experiment succeeds.

I created a file test_input with the following lines:

1 hello
2 test
3 abacus
4 qwerty
5 what
6 dvorak

With this in place, my goal was to be able to run something like cat test_file | parser_concept. After a bit of searching I found the lein-exec plugin for Leiningen, and after very minor setup I was able to start iterating with input piped in from elsewhere.

The first step was to open my profiles.clj file in my ~/.lein directory. I made sure lein-exec was specified in my user plugins as so:

{:user {:plugins [[lein-exec "0.2.1"]
                  ;other plugins for lein
                 ]}}

With this in place I just put the following line at the top of my script.clj file:

#!/usr/bin/env lein exec

I then changed the permissions of script.clj file to make it executable, I was able to run the following and have my code run against the input.

cat test_input | ./script.clj

I will be posting a follow up entry outlining my next step of experimenting with “processing” each line read in as a future.

Remove first and last lines from file in OS X

Just a quick post to help burn this into longer term memory.

Today I was having to check some info in a generated csv file that had a header and footer row. I only wanted the records in between, so I needed to remove the first and last lines of that CSV, after I got the columns I needed.

cut <args> my.csv | tail -n +2 | sed '$d'

The tail -n +2 command starts at the second line and outputs the input stream/file. The sed '$d' command deletes the last line of the file.

Setting the title in Terminal on OS X

Working through my Project Euler problems in Clojure, I wound up having a number of Terminal windows open at the same time: one for the REPL, one for a prompt to commit to my Project Euler git repo, one for a Midje program, one to start the Light Table Playground, and various others with other things I had going on.

With all of these windows, getting stacked on top of one another I was losing track of which Terminal window was which when I went to the Window menu for Terminal to get to the window I wanted.

I found this post Mac OS X Change the Terminal Window Title and then followed the instructions to get a nice little shell program setup to set the title of the window.

echo -n -e "33]0;$107"

Where I use the $1 instead the title itself, or a environment variable, per his two examples, to be able to have the first argument to the script, so now I can just run:

title "Light Table Playground"

I just figured I would share my find for anyone out there who hasn’t yet stumbled across it yet.

–Proctor