Monday, October 18, 2010

Lesson 9 - Multimethods and routing for our server

In Lesson 6 we saw how to parse a HTTP Request. In Lesson 7 we learned to send a proper response. For our server to really do something it needs to handle varied requests. So in this lesson we will learn to respond to requests for the index page ("/"), an about page ("/about"), a contact page ("/contact") and send a Not Found error for any other request.

To achieve the above we need to route the requests to the appropriate handler functions or otherwise called controllers. In traditional object oriented programming you would have created a base "Controller" class and created extensions to handle each type of request. In a functional programming language like Clojure we can achieve the same results using multimethods.

First let us write a route map for our program.

(def routes {
  "/" "index",
  "/about" "about",
  "/contact" "contact"
})
Here we define a map called routes. Each key in the map is the request path we want to handle, which is mapped to a corresponding string.

Next we define a multimethod with defmulti and name it controller.

(defmulti controller
  (fn [request]
    (routes (request :path))))
defmulti takes a function as its argument and based on the value returned it will call the appropriate function. In our case we pass it an anonymous function which takes a request object as its argument. The request object is the same parsed request we saw in Lesson 6. First we extract the path from the request (request :path). Then using the path as key in our routes map we return the corresponding string for the path.

Now how will it call the appropriate function? Here is our function to handle the index page.

(defmethod controller "index" [request]
  (send-response
        (assoc html-response :body "This is the index page")))
We have to define each of our required functions with defmethod. It takes the same name as the defmulti, but the next argument after the name is the dispatch-val. For this particular function to be called the function in defmulti must return a value equal to this dispatch-val. In our case the dispatch val is the string "index" which corresponds to our "/" path. The rest of the definition is the same as any other function you define.

Here are our other handler functions.
(defmethod controller "about" [request]
  (send-response
        (assoc html-response :body "This is the about page")))

(defmethod controller "contact" [request]
  (send-response
        (assoc html-response :body "This is the contact page")))
  
(defmethod controller :default [request]
  (println "HTTP/1.0 404 Not Found"))
Notice the last one. A dispatch-val of :default indicates that this is the default function in case no suitable function was found.

Here is the complete source code.
(use 'clojure.contrib.server-socket)
(use '[clojure.string :only (join split)])
(import  '(java.io BufferedReader InputStreamReader PrintWriter))

(def routes {
  "/" "index",
  "/about" "about",
  "/contact" "contact"
})

(defn parse-request []
  (loop
    [ result
        (zipmap
          [:method :path :protocol]
          (split (read-line) #"\s"))
      line (read-line)]
    (if (empty? line)
      result
      (recur
        (assoc
          result
          (keyword (first (split line #":\s+")))
          (last (split line #":\s+")))
        (read-line)))))
         
(def html-response
  { :status-line "HTTP/1.0 200 OK",
    :headers {:Content-Type "text/html"}})
            
(defn send-response [response]
  (let [headers (assoc (response :headers)
                  :Content-Length (count (response :body)))]
    (println (response :status-line))
    (println
      (join
        (for [key (keys headers) :let [value (headers key)]]
          (format "%s: %s\n" (name key) value))))
    (print (response :body))))

(defmulti controller
  (fn [request]
    (routes (request :path))))

(defmethod controller "index" [request]
  (send-response
        (assoc html-response :body "This is the index page")))

(defmethod controller "about" [request]
  (send-response
        (assoc html-response :body "This is the about page")))

(defmethod controller "contact" [request]
  (send-response
        (assoc html-response :body "This is the contact page")))
  
(defmethod controller :default [request]
  (println "HTTP/1.0 404 Not Found"))

(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (controller (parse-request))
      (flush))))



Tuesday, October 5, 2010

Lesson 8 - Macros! Macros! Everywhere

We saw anonymous functions in Lesson 4. Here is a simple anonymous function.
(fn [name] (println "Hello" name))
The problem with the above for me, coming after using javascript is that "fn" is too cryptic for my liking. I would have liked to have done
(function [name] (println "Hello" name))
I would rather write "function", which is more verbose, but makes code easier to understand for people coming to clojure from other languages. Now if I had some way, to kind of magically transform the code in the second line using "function" form to the first line using "fn", before the code is actually executed, I could have my way!

That is exactly what the defmacro function does. It takes a macro definition in the code, and replaces it with executable code at compile time. Macro's are like templates. It so happens that they generate executable code! This is what makes Macro's in Clojure and other dialects of Lisp so powerful, and so different to Macro's in other languages. Here is our simple trivial macro.

(defmacro function [args & body]
  `(fn ~args ~@body))

The defmacro function takes a name, "function" in this case and set of arguments. In this case the first argument is args, and the second "& body" represents a variable number of arguments. The backtick "`" indicates that "(fn ~args ~@body)" should not be evaluated. Instead what happens is that at compile time new code is generated to replace the macro with the "~" tilde arguments replaced with the actual arguments. So when the compiler encounters
(function [name] (println "Hello" name))
it replaces "~args" with "[name]" and "~@body" with whatever the rest of the arguments are, in this case "(println "Hello" name)". So the new code replacing the whole macro will be
(fn [name] (println "Hello" name))
which is what we wanted to begin with.

It is important to note that the arguments you passed to defmacro "[name] (println "Hello" name)" are not evaluated and are passed verbatim to the template. If you had passed the same arguments to a function on the other hand they would have been evaluated immediately.

Now let us actually execute some code.


(defmacro function [args & body]
  `(fn ~args ~@body))


((function [name]
  (println "Hello" name)) "World!")

What we are doing above is creating an anonymous function using the "function" macro and calling that function immediately with argument "World!". To execute an anonymous function immediately simply pass the function as the first argument of a list with the rest of the arguments in the list being the arguments to be passed to the function. The point to note above is that the variable "name" passed to the anonymous function is only visible within the function body .ie. it is a lexically scoped variable, just like the variables created with the "let" function we saw in lesson 7.

(defmacro function [args & body]
  `(fn ~args ~@body))

(println (macroexpand-1 '(function [name]
  (println "Hello" name))))

Run the program above. The macroexpand-1 function takes a quoted' macro and expands it to its executable form. In this case the expansion is
(clojure.core/fn [name] (println Hello name))

The defn function which we used for defining functions is also a macro. Here is a simple defn macro using our own "function" macro.

(defmacro function [args & body]
  `(fn ~args ~@body))
  
(defmacro define-function [fn-name args & body]
  `(def ~fn-name (function ~args ~@body)))

(define-function say-hello [name]
  (println "Hello" name))

(say-hello "World!")

Run the above code. So we created two macro's. "function" works exactly like "fn". And "define-function" works exactly like "defn".

Next we take on Multimethods and routing for our server.

Friday, October 1, 2010

Lesson 7 - What? No Variables in Clojure?

If you noticed by now, we have gone through six lessons and haven't yet created a variable. At least not explicitly. We did create some lexically scoped variables that were created implicitly as function arguments, or bindings to the "loop" and "reduce" functions we saw. It is highly unlikely that you would go through six lessons in any programming language (non Lisp) and not create a variable.

There are global variables in Clojure, which you create with the "def" function. And the "let" function which allows you to create lexically scoped variables. However the fact that we haven't used them so far is a good thing. The more variables you have in your program, the more you have to keep track of them. And the more the chances of inadvertently introducing a bug.

Also in Clojure you can get away with not creating variables because of the usage of higher order functions. Higher order functions are functions that take other functions as arguments. You already saw the higher order function "reduce" in action.

Clojure variables are immutable. You cannot modify the values after creation. You can rebind a variable name to another value though. This is a feature of Clojure not a bug! There are good reasons for this, concurrency being one of them.

Let us get on with improving our web server.



(use 'clojure.contrib.server-socket)
(use '[clojure.string :only (join)])
(import  '(java.io BufferedReader InputStreamReader PrintWriter))
    
(def default-response
  { :status-line "HTTP/1.0 200 OK",
    :headers {:Content-Type "text/html"},
    :body "<h1>A Web Server written in Clojure</h1>"})
  
(defn send-response [response]
  (let [headers (assoc (response :headers)
                  :Content-Length (count (response :body)))]
    (println (response :status-line))
    (println
      (join
        (for [header (keys headers)]
          (format "%s: %s\n" (name header) (headers header)))))
    (print (response :body))))

(create-server
  8080
  (fn [in out]
    (binding

      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (send-response default-response)
      (flush))))



Run the above code and browse to localhost:8080. What we have done here is set up a default response for our web server. We define a global variable default-response, which is a map structure with three keys and their corresponding values. Note that the value of the ":headers" key is itself another map.

We then define the send-response function which writes to the output stream, a given response map. The first thing the function does is define a lexically scoped variable called headers.


  (let [headers (assoc (response :headers)
                  :Content-Length (count (response :body)))]


The let function takes an array of bindings. Here there is only one binding. The variable "headers" is bound to the return value of the assoc function. The assoc function takes the :headers map of the response and adds a :Content-Length key, and its value, is returned by the count function which returns the length of the :body value of the response. The visibility of this bound "headers" variable is the body of the "let" function. Their are three "println" statements in this function. Let us look at the second one.

(join
  (for [header (keys headers)]
    (format "%s: %s\n" (name header) (headers header)))))

Let us look at the "for" statement first. It takes the collection of keys returned by (keys headers) and binds each value to the "header" variable and executes the "format" function for each header key. The format string takes two parameters, a string value of the header key (name header) and the value of the header (headers header) and replaces these two values in the format string. The collection of header lines returned by "for" is then joined by the "join" function.

In this case the output of the function will be

HTTP/1.0 200 OK
Content-Type: text/html
Content-Length: 40


<h1>A Web Server written in Clojure</h1>


We tackle Macro's in the next Lesson.

Wednesday, September 29, 2010

Lesson 6 - Parsing the HTTP Request in Clojure

In our previous post we managed to dump the http request headers back to the output to be viewed in the browser. If we want our server to do any better than what it does so far, we need to parse the request headers into a clojure data structure so we can query the various components of the request. If you are not familiar with a http request have a look at your last programs output again. The first line of the http request has three parts. A method "GET", a path "/", and the protocol "HTTP/1.1". The rest of the lines are name value pairs separated by a ":". We will parse these strings to a clojure key value map and dump the output to our browser.


(use '[clojure.contrib.str-utils2 :only (split)])
(use 'clojure.contrib.server-socket)
(import  '(java.io BufferedReader InputStreamReader PrintWriter))



(defn process-request-first-line []
  (zipmap
    [:method :path :protocol]
    (split (read-line) #"\s")))


(defn process-request []
  (loop
    [ result (process-request-first-line)
      line (read-line)]
    (if (empty? line)
      result
      (recur
        (assoc
          result
          (keyword (first (split line #":\s+")))
          (last (split line #":\s+")))
        (read-line)))))


(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (println (process-request))
      (flush))))


Run the program above and open localhost:8080 in your browser. You should see a dump of a key value map. My output was something like this.
{:path /, :protocol HTTP/1.1, :Accept application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5, :Accept-Encoding gzip,deflate,sdch, :method GET, :User-Agent Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.55 Safari/534.3, :Accept-Charset ISO-8859-1,utf-8;q=0.7,*;q=0.3, :Host localhost:8080, :Accept-Language en-US,en;q=0.8, :Connection keep-alive}


If all the above code and output looks a bit cryptic at the moment, don't worry, I will explain everything by the end of this lesson. The output above is a closure map with keyword - value pairs
{:path "/", :protocol "HTTP/1.1" .... etc etc }
:path is the first keyword whose value is "/", next is :protocol with value "HTTP/1.1" and so on.


Now let us look at the new source code we added.
(use '[clojure.contrib.str-utils2 :only (split)])
We first use the clojure.contrib.str-utils2 split function in our namespace. We use the :only keyword to indicate that only the split function will be imported.


(defn process-request-first-line []
  (zipmap
    [:method :path :protocol]
    (split (read-line) #"\s")))


We then define a function called process-request-first-line which processes the first line of the http request and returns a key value map of the three components of the first line. The zipmap function takes a collection of keys [:method :path :protocol] and a collection of values returned by
(split (read-line) #"\s")
and returns a map with the keys mapped to the corresponding values. The split function reads the first line and splits it with the regular expression #"\s" which is the space character, into three values. In our case the returned map will be
{:method "GET" :path "/" :protocol "HTTP/1.1"}


Let us look at the process-request function. The function loops through all the lines in the HTTP Request and fills up the map with the keyword value pairs. First the loop function binds two lexically scoped variables "result" and "line". "result" is bound to the value returned by process-request-first-line, which is the key value map above. "line" is bound to the second line of the http request. Remember that the first line was read by process-request-first-line function in the previous binding.


With the binding done we go through the expressions in the loop function. First we check if "line" is empty. If yes then we are done. Return "result". If not the recur function is called. The recur function will now bind new values to "result" and "line" and pass control back to the top of the loop.


        (assoc
          result
          (keyword (first (split line #":\s+")))
          (last (split line #":\s+")))


The assoc function takes a map "result" and adds a new keyword value pair to it. In this case the keyword is the first part of "line" before the ":" in the line.
(split line #":\s+")
This line splits "line" into two parts based on the regular expression #":\s".
(first (split line #":\s+"))
The "first" function takes the first element of the split.
(keyword (first (split line #":\s+")))
The "keyword" function converts the "first" string returned into a keyword.
(last (split line #":\s+"))
The "last" function returns the last part of the string "line" which is the value of the keyword in the previous line.
So now "result" is bound with the original map with this new key value pair added. In the last
        (read-line)))))
recur binds "line" to the next line read and we go to the top of the loop.

We really did not need the process-request-first-line function. I had used that function to simplify understanding of the code. I will now combine both the function and you will see the final source code below.


(use '[clojure.contrib.str-utils2 :only (split)])
(use 'clojure.contrib.server-socket)
(import  '(java.io BufferedReader InputStreamReader PrintWriter))


(defn process-request []
  (loop
    [ result
        (zipmap
          [:method :path :protocol]
          (split (read-line) #"\s"))
      line (read-line)]
    (if (empty? line)
      result
      (recur
        (assoc
          result
          (keyword (first (split line #":\s+")))
          (last (split line #":\s+")))
        (read-line)))))


(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (println (process-request))
      (flush))))


Lets move on to learning about variables in Clojure

Tuesday, September 28, 2010

Lesson 5 - A Web Server in Clojure

In our previous lessons we generated some web pages in clojure. However we just viewed those pages in our browser without a web server. Now let us roll out our own web server in clojure to view our own shiny little web site.


(use 'clojure.contrib.server-socket)


(create-server
  8080
  (fn [in out]
    (binding
      [*out* (java.io.PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (println "<h1>Wooo hooo hooo, my first web server!</h1>")
      (flush))))

Type the code above into a file called server.clj and run it. The program will not exit the terminal. Never mind. Now open a browser window and browse to "http://localhost:8080". You just rolled your first web server!

Now for some explanation. We use the server-socket library from clojure.contrib. The create-server function takes two parameters a port number and a callback function to call when a connection is made. The server listens to post 8080 and when a connection is made, the callback function is called with two arguments, a socket input stream and a socket output stream.
In the following line
[*out* (java.io.PrintWriter. out)]
we bind the stdout stream "*out*" to an instance of a java.io.PrintWriter class created with the socket output stream we received. We then print to the stdout and flush it. Thats it!

I had mentioned earlier that clojure can also call java functions and here we see
(java.io.PrintWriter. out)
whose equivalent in clojure is
(new java.io.PrintWriter out)
and in java
new java.io.PrintWriter(out)

Our shiny new web server is pretty rudimentary (to say the least).  We dump the output to anything that connects, without checking the request. So lets fix that first.


(use 'clojure.contrib.server-socket)
(import  '(java.io BufferedReader InputStreamReader PrintWriter))

(create-server
  8080
  (fn [in out]
    (binding
      [ *in* (BufferedReader. (InputStreamReader. in))
        *out* (PrintWriter. out)]
      (println "HTTP/1.0 200 OK")
      (println "Content-Type: text/html")
      (println "")
      (loop [line (read-line)]
        (println (str line "<br/>"))
        (if-not (empty? line)
          (recur (read-line))))
      (flush))))



Run the above program and browse to localhost:8080. The program simply returns all the lines of text in the input stream back to the browser with a little html formatting.

First, we use the import statement to import a few java classes into our namespace so that we can just use the class names without the namespace. We also bind the stdin "*in*" to our socket input via BufferedReader which takes an argument InputStreamReader which in turn takes our socket input stream "in" as argument.

Now let us look at the following function.
(loop [line (read-line)]
  (println (str line "<br/>"))
  (if-not (empty? line)
    (recur (read-line))))

The loop function takes a set of bindings as its arguments, and hitting a recur statement within its scope will return control to the top of the loop. Here we bind the value of (read-line) which will be the first line of *in* to a lexically scoped variable named "line". Next we print line to *out*.  Next we call the if-not function which in turn calls the empty? function to check if not "line" is empty. If it is not empty then we call recur function which will (read-line) and rebind "line" to the new line we read and pass control back to the top of the loop.

In the next lesson we look at parsing the HTTP Request.

Sunday, September 26, 2010

Lesson 4 - Anonymous functions in Clojure

Ok, now that we have done a simple web page, let us add a table to that page. This table must be created from given rows of data as shown below in the example row.
["Bob" "bob@example.com" "29"]
Thats not all. We need to first convert this row of data into a structure required by the prxml function, before prxml can generate the actual xml.

(defn gen-row [row]
  (reduce
    (fn [arr elem]
      (conj
        arr
        [:td elem]))
    [:tr]
    row))


(println (gen-row ["Bob" "bob@example.com" "29"]))

Run this program and you should get the following line printed to your terminal.
[:tr [:td Bob] [:td bob@example.com] [:td 29]]
Ok. So this is what we wanted. Look at the last line of the code. We first call a function called println, which prints a string to the stdout adding a line feed to the end of the output. To println we pass a call to our own function called gen-row to which we pass our row vector. gen-row takes one vector as its argument and returns another vector structured for the prxml function.

The whole convertion from the given vector form to the returned form is carried out by the reduce function. reduce takes three parameters as its arguments. The first parameter is a function "(fn [arr elem] (conj arr [:td elem]))". The second parameter is an optional value which is "[:tr]" in this case. The third param is a collection. The reduce function will call the given function with the value and first element of the collection, and will repeat this for every element in the collection. The first function is interesting because we never defined it. It is an anonymous function. You call an anonymous function with "fn". This anonymous function takes two prameters. A vector arr and a string elem from the row array. It then calls the "conj' function, which takes a collection as its first argument and adds the second argument to the collection. In this case the first argument is our initial value array, to which a new element array is added in every iteration.

Now we can generate one row of the table as required by the prxml functions. But tables seldom contain only one row. So we will create a gen-table function which will be called with an array of multiple rows.


(defn gen-row [row]
  (reduce
    (fn [arr elem]
      (conj
        arr
        [:td elem]))
    [:tr]
    row))
    
(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (gen-row row)))
    [:table]
    rows))


(println
  (gen-table
    [
      ["Bob" "bob@example.com" "29"]
      ["Bill" "bill@example.com" "32"]
    ]))

Now run this program and the output you will get will be for the whole table. We really dont need a separate gen-row function so we will add it to gen-table and shorten our program.
  
(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (reduce
          (fn [arr elem]
            (conj
              arr
              [:td elem]))
          [:tr]
          row)))
    [:table]
    rows))


(println
  (gen-table
    [
      ["Bob" "bob@example.com" "29"]
      ["Bill" "bill@example.com" "32"]
    ]))

Try this and you should get the same earlier output. The anonymous function fn has a simple form, in which you can simply write the function as #( ... ) and call the passed arguments as %1 %2 etc. So lets shorten our program even more.

(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (reduce
          #(conj
            %1
            [:td %2])
          [:tr]
          row)))
    [:table]
    rows))


(println
  (gen-table
    [
      ["Bob" "bob@example.com" "29"]
      ["Bill" "bill@example.com" "32"]
    ]))
  
So there you are, an even shorter program. Notice that I only used the short form, only for one of the anonymous functions, because nested short forms of the anonymous functions are not allowed.

Now let us put all this together with our html page generator of lesson 3 and see if we can actually see the table on a page.


(use 'clojure.contrib.prxml)


(defn gen-table[rows]
  (reduce
    (fn [arr row]
      (conj
        arr
        (reduce
          #(conj
            %1
            [:td %2])
          [:tr]
          row)))
    [:table {:border "2"}]
    rows))


(defn generate-page [filename title content]
  (spit
    filename
    (with-out-str
      (binding [*prxml-indent* 4]
        (prxml
          [:html
            [:body
              [:h1  {:style "color:red"} title]
              (gen-table content)]])))))


(generate-page
  "table.html"
  "An html page with a table"
  [
    ["Bob" "bob@example.com" "29"]
    ["Bill" "bill@example.com" "32"]
  ])

Now run the program and open the generated "table.html" in a browser window. Aren't you excited about the possibilities of Clojure? Then dont wait, on to lesson 5 - A Web Server in Clojure!

Saturday, September 25, 2010

Lesson 3 - User defined functions in Clojure

In the last two lessons we saw a few Clojure functions provided by "clojure.core" and "clojure.contrib" libraries. In this lesson we will roll out our own function in Clojure.

(use 'clojure.contrib.prxml)
(spit
  "hello.html"
  (with-out-str
    (binding [*prxml-indent* 4]
      (prxml
        [:html
          [:body
            [:h1 "My Home Page"]]]))))
          
Type the above code into a file lesson3.clj and run it. By now you would have figured what the code does. It generates an html file called hello.html. Open the file in your browser. And view its source code in your browser. As you can see, each vector represents an XML element (in this case html element). A keyword element represents the tag name, and the second element represents the content of the parent element.

Now change the last line to read as follows
[:h1 {:style "color:red"} "My Home Page"]]]))))
Now run the program and reload hello.html in the browser and look at the source code. Here we have added an attribute to the h1 element. In prxml, element attributes are represented by maps. A map is a structure in which the keyword and value are wrapped in curly braces. You can have multiple keyword values in a map.

Let us add a content part to the page.

(use 'clojure.contrib.prxml)
(spit
  "hello.html"
  (with-out-str
    (binding [*prxml-indent* 4]
      (prxml
        [:html
          [:body
            [:h1  {:style "color:red"} "My Home Page"]
            [:div "This is the content of my page"]]]))))
          
Run the program and view source again, to make sure everything is as expected. Now let us say we want to generate more pages for our shiny new little website. Some parts of the page will remain consistent in all the pages we generate. The only parts that change will be the file name, title and content of each page. Right! So instead of copying the code for each page we can write a function to generate every page given the variable parameters for each page.

(use 'clojure.contrib.prxml)
(defn generate-page [filename title content]
  (spit
    filename
    (with-out-str
      (binding [*prxml-indent* 4]
        (prxml
          [:html
            [:body
              [:h1  {:style "color:red"} title]
              [:div content]]])))))
            
(generate-page "about.html" "About Me page" "This content is about Me")

Now run this program, open "about.html" in your browser and view the source. Congratulations! You have done your first Clojure user defined function. You define functions in Clojure using defn. defn takes a function name, a set of params in a vector, and a function body as its arguments. defn also takes two more optional arguments doc-string? and attr-map?, which we will cover in another lesson. In the last line we called the generate-page function passing it the three required parameters.