Lesson 6 - Parsing the HTTP Request in Clojure

In our previous post we managed to dump the http request headers back to the output to be viewed in the browser. If we want our server to do any better than what it does so far, we need to parse the request headers into a clojure data structure so we can query the various components of the request. If you are not familiar with a http request have a look at your last programs output again. The first line of the http request has three parts. A method "GET", a path "/", and the protocol "HTTP/1.1". The rest of the lines are name value pairs separated by a ":". We will parse these strings to a clojure key value map and dump the output to our browser.


(use '[clojure.contrib.str-utils2 :only (split)])
(use 'clojure.contrib.server-socket)
(import  '(java.io BufferedReader InputStreamReader PrintWriter))



(defn process-request-first-line []
  (zipmap
    [:method :path :protocol]
    (split (read-line) #"\s")))


(defn process-request []
  (loop
    [ result (process-request-first-line)
      line (read-line)]
    (if (empty? line)
      result
      (recur
        (assoc
          result
          (keyword (first (split line #":\s+")))
          (last (split line #":\s+")))
        (read-line)))))


(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (println (process-request))
      (flush))))


Run the program above and open localhost:8080 in your browser. You should see a dump of a key value map. My output was something like this.
{:path /, :protocol HTTP/1.1, :Accept application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5, :Accept-Encoding gzip,deflate,sdch, :method GET, :User-Agent Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.55 Safari/534.3, :Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.3, :Host localhost:8080, :Accept-Language en-US,en;q=0.8, :Connection keep-alive}


If all the above code and output looks a bit cryptic at the moment, don't worry, I will explain everything by the end of this lesson. The output above is a closure map with keyword - value pairs
{:path "/", :protocol "HTTP/1.1" .... etc etc }
:path is the first keyword whose value is "/", next is :protocol with value "HTTP/1.1" and so on.


Now let us look at the new source code we added.
(use '[clojure.contrib.str-utils2 :only (split)])
We first use the clojure.contrib.str-utils2 split function in our namespace. We use the :only keyword to indicate that only the split function will be imported.


(defn process-request-first-line []

  (zipmap

    [:method :path :protocol]

    (split (read-line) #"\s")))

We then define a function called process-request-first-line which processes the first line of the http request and returns a key value map of the three components of the first line. The zipmap function takes a collection of keys [:method :path :protocol] and a collection of values returned by
(split (read-line) #"\s")
and returns a map with the keys mapped to the corresponding values. The split function reads the first line and splits it with the regular expression #"\s" which is the space character, into three values. In our case the returned map will be
{:method "GET" :path "/" :protocol "HTTP/1.1"}


Let us look at the process-request function. The function loops through all the lines in the HTTP Request and fills up the map with the keyword value pairs. First the loop function binds two lexically scoped variables "result" and "line". "result" is bound to the value returned by process-request-first-line, which is the key value map above. "line" is bound to the second line of the http request. Remember that the first line was read by process-request-first-line function in the previous binding.


With the binding done we go through the expressions in the loop function. First we check if "line" is empty. If yes then we are done. Return "result". If not the recur function is called. The recur function will now bind new values to "result" and "line" and pass control back to the top of the loop.

        (assoc

          result

          (keyword (first (split line #":\s+")))

          (last (split line #":\s+")))

The assoc function takes a map "result" and adds a new keyword value pair to it. In this case the keyword is the first part of "line" before the ":" in the line.
(split line #":\s+")
This line splits "line" into two parts based on the regular expression #":\s".
(first (split line #":\s+"))
The "first" function takes the first element of the split.
(keyword (first (split line #":\s+")))
The "keyword" function converts the "first" string returned into a keyword.
(last (split line #":\s+"))
The "last" function returns the last part of the string "line" which is the value of the keyword in the previous line.
So now "result" is bound with the original map with this new key value pair added. In the last
        (read-line)))))
recur binds "line" to the next line read and we go to the top of the loop.

We really did not need the process-request-first-line function. I had used that function to simplify understanding of the code. I will now combine both the function and you will see the final source code below.

(use '[clojure.contrib.str-utils2 :only (split)])
(use 'clojure.contrib.server-socket)
(import  '(java.io BufferedReader InputStreamReader PrintWriter))


(defn process-request []
  (loop
    [ result
        (zipmap
          [:method :path :protocol]
          (split (read-line) #"\s"))
      line (read-line)]
    (if (empty? line)
      result
      (recur
        (assoc
          result
          (keyword (first (split line #":\s+")))
          (last (split line #":\s+")))
        (read-line)))))


(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (println (process-request))
      (flush))))


Lets move on to learning about variables in Clojure

Lesson 5 - A Web Server in Clojure

In our previous lessons we generated some web pages in clojure. However we just viewed those pages in our browser without a web server. Now let us roll out our own web server in clojure to view our own shiny little web site.


(use 'clojure.contrib.server-socket)


(create-server
  8080
  (fn [in out]
    (binding
      [*out* (java.io.PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (println "<h1>Wooo hooo hooo, my first web server!</h1>")
      (flush))))

Type the code above into a file called server.clj and run it. The program will not exit the terminal. Never mind. Now open a browser window and browse to "http://localhost:8080". You just rolled your first web server!

Now for some explanation. We use the server-socket library from clojure.contrib. The create-server function takes two parameters a port number and a callback function to call when a connection is made. The server listens to post 8080 and when a connection is made, the callback function is called with two arguments, a socket input stream and a socket output stream.
In the following line
[*out* (java.io.PrintWriter. out)]
we bind the stdout stream "*out*" to an instance of a java.io.PrintWriter class created with the socket output stream we received. We then print to the stdout and flush it. Thats it!

I had mentioned earlier that clojure can also call java functions and here we see
(java.io.PrintWriter. out)
whose equivalent in clojure is
(new java.io.PrintWriter out)
and in java
new java.io.PrintWriter(out)

Our shiny new web server is pretty rudimentary (to say the least).  We dump the output to anything that connects, without checking the request. So lets fix that first.


(use 'clojure.contrib.server-socket)
(import  '(java.io BufferedReader InputStreamReader PrintWriter))

(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (loop [line (read-line)]
        (println (str line "<br/>"))
        (if-not (empty? line)
          (recur (read-line))))
      (flush))))


Run the above program and browse to localhost:8080. The program simply returns all the lines of text in the input stream back to the browser with a little html formatting.

First, we use the import statement to import a few java classes into our namespace so that we can just use the class names without the namespace. We also bind the stdin "*in*" to our socket input via BufferedReader which takes an argument InputStreamReader which in turn takes our socket input stream "in" as argument.

Now let us look at the following function.
(loop [line (read-line)]
  (println (str line "<br/>"))
  (if-not (empty? line)
    (recur (read-line))))

The loop function takes a set of bindings as its arguments, and hitting a recur statement within its scope will return control to the top of the loop. Here we bind the value of (read-line) which will be the first line of *in* to a lexically scoped variable named "line". Next we print line to *out*.  Next we call the if-not function which in turn calls the empty? function to check if not "line" is empty. If it is not empty then we call recur function which will (read-line) and rebind "line" to the new line we read and pass control back to the top of the loop.

In the next lesson we look at parsing the HTTP Request.

Lesson 4 - Anonymous functions in Clojure

Ok, now that we have done a simple web page, let us add a table to that page. This table must be created from given rows of data as shown below in the example row.
["Bob" "bob@example.com" "29"]
Thats not all. We need to first convert this row of data into a structure required by the prxml function, before prxml can generate the actual xml.

(defn gen-row [row]
  (reduce
    (fn [arr elem]
      (conj
        arr
        [:td elem]))
    [:tr]
    row))


(println (gen-row ["Bob" "bob@example.com" "29"]))

Run this program and you should get the following line printed to your terminal.
[:tr [:td Bob] [:td bob@example.com] [:td 29]]
Ok. So this is what we wanted. Look at the last line of the code. We first call a function called println, which prints a string to the stdout adding a line feed to the end of the output. To println we pass a call to our own function called gen-row to which we pass our row vector. gen-row takes one vector as its argument and returns another vector structured for the prxml function.

The whole convertion from the given vector form to the returned form is carried out by the reduce function. reduce takes three parameters as its arguments. The first parameter is a function "(fn [arr elem] (conj arr [:td elem]))". The second parameter is an optional value which is "[:tr]" in this case. The third param is a collection. The reduce function will call the given function with the value and first element of the collection, and will repeat this for every element in the collection. The first function is interesting because we never defined it. It is an anonymous function. You call an anonymous function with "fn". This anonymous function takes two prameters. A vector arr and a string elem from the row array. It then calls the "conj' function, which takes a collection as its first argument and adds the second argument to the collection. In this case the first argument is our initial value array, to which a new element array is added in every iteration.

Now we can generate one row of the table as required by the prxml functions. But tables seldom contain only one row. So we will create a gen-table function which will be called with an array of multiple rows.


(defn gen-row [row]
  (reduce
    (fn [arr elem]
      (conj
        arr
        [:td elem]))
    [:tr]
    row))
    
(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (gen-row row)))
    [:table]
    rows))


(println
  (gen-table
    [
      ["Bob" "bob@example.com" "29"]
      ["Bill" "bill@example.com" "32"]
    ]))

Now run this program and the output you will get will be for the whole table. We really dont need a separate gen-row function so we will add it to gen-table and shorten our program.
  
(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (reduce
          (fn [arr elem]
            (conj
              arr
              [:td elem]))
          [:tr]
          row)))
    [:table]
    rows))


(println
  (gen-table
    [
      ["Bob" "bob@example.com" "29"]
      ["Bill" "bill@example.com" "32"]
    ]))

Try this and you should get the same earlier output. The anonymous function fn has a simple form, in which you can simply write the function as #( ... ) and call the passed arguments as %1 %2 etc. So lets shorten our program even more.

(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (reduce
          #(conj
            %1
            [:td %2])
          [:tr]
          row)))
    [:table]
    rows))


(println
  (gen-table
    [
      ["Bob" "bob@example.com" "29"]
      ["Bill" "bill@example.com" "32"]
    ]))
  
So there you are, an even shorter program. Notice that I only used the short form, only for one of the anonymous functions, because nested short forms of the anonymous functions are not allowed.

Now let us put all this together with our html page generator of lesson 3 and see if we can actually see the table on a page.


(use 'clojure.contrib.prxml)


(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (reduce
          #(conj
            %1
            [:td %2])
          [:tr]
          row)))
    [:table {:border "2"}]
    rows))


(defn generate-page [filename title content]
  (spit
    filename
    (with-out-str
      (binding [*prxml-indent* 4]
        (prxml
          [:html
            [:body
              [:h1  {:style "color:red"} title]
              (gen-table content)]])))))


(generate-page
  "table.html"
  "An html page with a table"
  [
    ["Bob" "bob@example.com" "29"]
    ["Bill" "bill@example.com" "32"]
  ])

Now run the program and open the generated "table.html" in a browser window. Aren't you excited about the possibilities of Clojure? Then dont wait, on to lesson 5 - A Web Server in Clojure!

Lesson 3 - User defined functions in Clojure

In this lesson we will roll out our own function in Clojure.

We'll use spit and with-out-str to write to file. But instead of emit, we'll use the emit-element function for our tags.

Type out the following code and save it as hellohtml.clj.

//hellohtml.clj

(use 'clojure.xml)
(println
 (with-out-str
  (emit-element
   {:tag :html :content
    [{:tag :body :content
     [{:tag :h1 :attrs {:style "color:red"} :content["My Homepage"]}
      {:tag :div :attrs {:style "text-align:center"} :content["This is the content"]} ]}]})))

Run it using lein exec

$ lein exec hellohtml.clj
<html>
<body>
<h1 style='color:red'>
My Homepage
</h1>
<div style='text-align:center'>
This is the content
</div>
</body>
</html>

If you compare it to the helloxmlfile.clj from Lesson 2, you should have an idea of what the above code does.

I know what you're thinking. SO MANY BRACKETS! Relax, you'll be fine.

What do we have here? HTML wasted on stdout. Let's save that to file by replacing println with spit <filename>.

//hellohtml.clj
(use 'clojure.xml)
(spit "hellohtml.html"
 (with-out-str
  (emit-element
   {:tag :html :content
    [{:tag :body :content
     [{:tag :h1 :attrs {:style "color:red"} :content["My Homepage"]}
      {:tag :div :attrs {:style "text-align:center"} :content["This is the content"]} ]}]})))

Run it using lein exec.

As expected we have a file called hellohtml.html with the code that was earlier sent to stdout.

Now let's generate more pages for our fictional website. We're going to be consistent with our minimalist design and the only parts that will change are the filename, title and the content. Since that's the case, why repeat the code for each page when you can roll your own function to do exactly that.

Let's create a new file called hellofunchtml.clj and modify our exisiting code so it looks like this.

//hellofunchtml.clj
(use 'clojure.xml)
(defn generate-page [filename title content]
  (spit filename
   (with-out-str
     (emit-element
       {:tag :html :content
        [{:tag :body :content
         [{:tag :h1 :attrs {:style "color:red"} :content[title]}
          {:tag :div :attrs {:style "text-align:center"} :content[content]} ]}]}))))

(generate-page "about.html" "About Me" "There's nothing to say.")
(generate-page "contact.html" "Contact Me" "Here's my email and phone. Spam me.")
(generate-page "faq.html" "FAQ" "Why was six afraid of seven? It wasn't. Numbers are not sentient and thus incapable of feeling fear.")
(generate-page "index.html" "Index" "Welcome to my homepage")

Run it using lein exec
Congratulations! You define functions in Clojure using defn. defn takes a function name, a set of params in a vector, and a function body as its arguments. defn also takes two more optional arguments doc-string? and attr-map?, which we will cover in another lesson. In the last lines we called the generate-page function passing it the three required parameters.

Go on to Lesson 4 - Anonymous functions in Clojure

Lesson 2 - Clojure Functions

In Lesson 1 we saw two Clojure functions in action. Spit and eval. Clojure comes with a set of functions available to you by default. These functions reside in the "clojure.core" library. Have a quick look at the available functions in clojure.core at the link given below and come back here.
https://clojure.github.io/clojure/clojure.core-api.html#clojure.core

Did you notice "+" "-" "/" "*" among the available functions? Yep! These are functions in Clojure. Try the following function in the clojure REPL. (To run the REPL just run clojure without any arguments).
(+ 1 2)

In Clojure every thing you do is with functions and functions only. In addition to the Clojure core functions you have the clojure.contrib library, with additional functions contributed by a bunch of nice people. Then you have third party libraries for doing various stuff. And thats not all. You have the whole load of java functions available to you in Clojure.

We're going to output some XML to stdout using emit.

Type (emit "hello") in a file called helloxmlfile.clj and run it using lein exec.

//helloxmlfile.clj
(emit "hello")

and

$ lein exec helloxmlfile.clj

BOOM! Wall of errors.

Scroll up through the wall and you'll see some English that might make some sense.
"Unable to resolve symbol: emit in this context"

What this means is that the Clojure compiler couldn't understand what or who you were referring to when you said 'emit'.

The emit function is part of the namespace clojure.xml, which is not imported in the lein REPL. You could use spit and println in the previous examples because they're part of the clojure.core namespace, which is imported when the lein REPL starts up. (Fun exercise: Type (all-ns) in the lein REPL to see all the available namespaces. Don't bother if the output doesn't make any sense now.)

So, we're going to tell our Clojure compiler where to look for, when we say emit, by using 'use'.

// helloxmlfile.clj
(use 'clojure.xml)
(emit "hello")

and

$ lein exec helloxml.clj

<?xml version='1.0' encoding='UTF-8'?>
hello

No error wall! Sweet. But, that's not enough XML.

Make your helloxmlfile.clj look like this and run lein exec again.

///helloxmlfile.clj
(use 'clojure.xml)
(emit {:tag :parent :attrs {:value "Hello"}
:content[
{:tag :child :attrs {:value "World"}}
{:tag :child :attrs {:value "Clojure"}}
]})

and

$ lein exec helloxml.clj

<?xml version='1.0' encoding='UTF-8'?>
<parent value='Hello'>
<child value='World'/>
<child value='Clojure'/>
</parent>

The parent and child elements come from a map, a data structure in Clojure, with alternating keys (:keys) and values. A key can be a value. A map can also be a value. The content is a vector that contains a map. We'll delve into keywords and vectors and maps later.

What we have now is some XML that is wasted on stdout. Let's save that to a file using what we've learnt earlier.

Combine spit and with-out-str to make your program look like this.

///helloxmlfile.clj
(use 'clojure.xml)
(spit "hello.xml" (with-out-str (emit {:tag :parent :attrs {:value "Hello"}
:content[
{:tag :child :attrs {:value "World"}}
{:tag :child :attrs {:value "Clojure"}}
]})))

And run it with lein exec.

$ lein exec helloxml.clj

There's no output on the screen. That's excellent because if you've been paying attention, we were outputting XML to file.

Check the contents of hello.xml.

$ cat hello.xml
<?xml version='1.0' encoding='UTF-8'?>
<parent value='Hello'>
<child value='World'/>
<child value='Clojure'/>
</parent>

Well done.

Alright! To sum it up, we just learnt to send the output of our Clojure program to file. Writing XML. A bit about namespaces. Using 'use'.

Lets roll out our own Clojure function in the next lesson.

Installing Clojure 1.2

To install Clojure 1.2 create a folder for clojure files in your file system. Lets call the folder clojure. Download the following files into the folder.
clojure-1.2.0.jar from here http://build.clojure.org/releases/org/clojure/clojure/1.2.0/
and
clojure-contrib-1.2.0.jar from here. http://build.clojure.org/releases/org/clojure/clojure-contrib/1.2.0/

Clojure requires java JDK so if you don't have it installed then you can download it here.
http://www.oracle.com/technetwork/java/javase/downloads/index.html

For Linux, create a shell script called clojure with the following lines and put it in your command path.

#!/bin/sh

java -cp "path/to/clojure/*:$PWD" clojure.main "$@"

For windows create a bat file called clojure.bat with the following content and set your windows path to the clojure folder.

@echo off
java -cp "path\to\clojure\*;." clojure.main %*

Lesson 1 - Hello Clojure

(spit "hello.txt" "Hello World! Hello Clojure")

Type the above code into a text editor and save it as "hello.clj". Now open a terminal command/window and run the program by typing "lein exec hello.clj" into the command line. You will see a file called "hello.txt" created in the same folder, with "Hello World! Hello Clojure" as its contents. Thats your first clojure program!

Of course you need to have Clojure, lein and lein exec installed on your computer, otherwise the "lein exec hello.clj" command will not work. If you don't have all of that installed, here are some instructions to get you upto speed.
http://fasttrackclojure.blogspot.in/2015/02/installing-clojure.html

Now let us look at the code. We see a Clojure expression in the parenthesis, whose first element is a function called spit. spit takes two arguments, a file name "hello.txt" and string which is the content of the file to be written (spit into).  You can see the documentation of the spit function here. http://clojure.github.com/clojure/clojure.core-api.html#clojure.core/spit.

The above expression is called a list, and the expression is evaluated immediately on execution. This list is also a function call. In Clojure a function call is just a list whose first element resolves to a function. Now try this

'(spit "hello.txt" "Hello World! Hello Clojure")

Add a quote ' to the beginning of the list and run your program again. (Delete the hello.txt file before executing your program). Now see what happens. Nothing. The list expression is not evaluated now. Your list expression is a piece of data now.

Lets do this now.

(eval
  '(spit "hello.txt" "Hello World! Hello Clojure"))

Bam! Your quoted piece of data is evaluated again! We wrapped the quoted expression in another list, which calls a function called eval, and we passed our quoted list as an argument to it. Here is the doc for the eval function. http://clojure.github.com/clojure/clojure.core-api.html#clojure.core/eval.

Your code can be data and your data can be code. Welcome to Lisp! Clojure is a language based on Lisp. And we are already beginning to see the power and simplicity of the language.

Are you already excited about Clojure like I am? Then go to Lesson 2.