dependencies
| (this space intentionally left almost blank) | |||||||||||||||||||||||||||||||||||||||
namespaces
| ||||||||||||||||||||||||||||||||||||||||
Overview This workshop is organized in numerical order of progression. Start with ch01 and move through each exercise (x0n) all the way through chapter 04/x04. Have fun!! | (ns clojure-north-2020.core) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x01-data) | ||||||||||||||||||||||||||||||||||||||||
A Clojure Application Starts as DataPerhaps the most overlooked and powerful aspect of Clojure is that it is a first-class data oriented language. That, combined with functional programming, sets it aside from nearly any other language. In this namespace, were going to consider the first step you should undertake when writing any Clojure program - modeling your domain as data. Forget your functions, code, etc. Instead just think about your domain. How might you model it as data? In Clojure you have 4 key data structures:
Given these simple literal data structures you should be able to model any problem in any domain. Model actual cases. This is a far more powerful technique than starting with a schema or class hierarchy. Consider how you might model the following domains:
For this project, we'll be modeling superheroes. Consider how you might model a superhero with attributes such as name, alias, powers, weapons, etc. One interesting aspect of this problem is how we reference other entities, such as villains/nemeses. These are also top-level items, so perhaps it makes sense to ID them by name vs. a plain old string. | ||||||||||||||||||||||||||||||||||||||||
Exercise: Model at least 2 superheros as data. For example, Batman may have the following characteristics:
How might you model referential relations, such as nemeses, alliances, or familial relationships? | (def data []) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x01-data-solutions) | ||||||||||||||||||||||||||||||||||||||||
(def data
[{:name "Batman"
:alias "Bruce Wayne"
:powers #{"Rich"}
:weapons #{"Utility Belt" "Kryptonite Spear"}
:alignment "Chaotic Good"
:nemesis [{:name "Joker"}
{:name "Penguin"}]}
{:name "Superman"
:alias "Clark Kent"
:powers #{"Strength" "Flight" "Bullet Immunity"}
:alignment "Lawful Good"
:nemesis [{:name "Lex Luthor"}
{:name "Zod"}
{:name "Faora"}]}
{:name "Wonder Woman"
:alias "Diana Prince"
:powers #{"Strength" "Flight"}
:weapons #{"Lasso of Truth" "Bracers"}
:alignment "Lawful Good"
:nemesis [{:name "Ares"}]}
{:name "Shazam"
:alias "Billy Batson"
:powers #{"Strength" "Bullet Immunity"}
:alignment "Neutral Good"
:nemesis [{:name "Dr. Thaddeus Sivana"}
{:name "Pride"}
{:name "Envy"}
{:name "Greed"}
{:name "Wrath"}
{:name "Sloth"}
{:name "Gluttony"}
{:name "Lust"}]}
{:name "Joker"
:alias "Jack Napier"
:alignment "Chaotic Evil"
:nemesis [{:name "Batman"}]
}
]) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x02-functions
(:require [clojure-csv.core :as csv]
[clojure-north-2020.ch01-data.x01-data :refer [data]]
[clojure-north-2020.ch01-data.x02-functions-solutions :as x02-soln]
[clojure.string :as cs]
[cuerdas.core :as cc])) | ||||||||||||||||||||||||||||||||||||||||
Manipulating Data and Building APIs with FunctionsAs a language, Clojure is great at working with data. It's very common once you establish a data model to work with it interactively to build domain API functions or just better understand the data. As you explore and find useful relations it is common to "lift" a form into a function. Exercise: Interact with our dataExplore the data generated in the last ns and write a useful function. Examples:
| ||||||||||||||||||||||||||||||||||||||||
Some Utility FunctionsIt is common to develop an data model and then bring in data from an external source such as a csv file. You will then wish to conform this csv data to your created internal model. We'll be doing this in the next few exercises so that we don't have to create data for hundreds of supers by hand. We will be reading in our data from csv files. Let's write some functions for converting our raw files into our data as we've modeled it previously. One of the great things about Clojure is that is effectively a DSL for data, so the functions we create now will be domain agnostic and of general utility in the future since we're just dealing with data. Exercise: Write a csv->data function
Step 1: Inspect the data. We can see that the first row is the column names and the rest are data. We also see some data quality issues:
| (comment
(->> "resources/heroes_information.csv"
slurp
csv/parse-csv
;Just look a a sampling of the data
(take 4))) | |||||||||||||||||||||||||||||||||||||||
Remove entries in a seq of pairs for which any of the following are true:
| (defn remove-bad-entries [m] ;TODO - implement) | |||||||||||||||||||||||||||||||||||||||
(comment
(= {:c "OK"}
(remove-bad-entries {nil "A" :a nil :b "" :c "OK"}))
(= [[:c "OK"]]
(remove-bad-entries [[nil "A"] [:a nil] [:b ""] [:c "OK"]]))) | ||||||||||||||||||||||||||||||||||||||||
Convert a sequence of vectors into a sequence of maps, assuming the first row of the vectors is a header row. | (defn table->maps [[headers & cols]] ;TODO - implement) | |||||||||||||||||||||||||||||||||||||||
(comment
(= [{:id 1 :name "Mark" :age 42}
{:id 2 :name "Sue" :age 12 :phone "123-456-7890"}
{:id 3 :name "Pat" :age 18}]
(table->maps
[["" "ID" "Name" "Age" "Phone"]
[0 1 "Mark" 42 "-"]
[1 2 "Sue" 12 "123-456-7890"]
[2 3 "Pat" 18 nil]]))) | ||||||||||||||||||||||||||||||||||||||||
(defn csv-file->maps [f]
(-> f
slurp
csv/parse-csv
;Once you get your solution in place, remove the external solution
x02-soln/table->maps)) | ||||||||||||||||||||||||||||||||||||||||
Other utility functionsThis won't be an exercise, but here are a few more utility functions that we will be using to parse our data. | ||||||||||||||||||||||||||||||||||||||||
Keywordize strings by:
| (defn kwize [s] (-> s (cs/replace #"\W+" " ") (cs/replace #"'" ) cc/keyword)) | |||||||||||||||||||||||||||||||||||||||
Update key k in map m with function f if there is a value in m for k. | (defn maybe-update [m k f] (cond-> m (some? (m k)) (update k f))) | |||||||||||||||||||||||||||||||||||||||
Bulk update several keys ks in map m with function f | (defn maybe-bulk-update [m ks f] (reduce (fn [m k] (maybe-update m k f)) m ks)) | |||||||||||||||||||||||||||||||||||||||
(comment
(maybe-bulk-update
{:width "2.1" :height "45.53" :name "Bob"}
[:width :height]
#(Double/parseDouble %))) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x02-functions-solutions
(:require [clojure-north-2020.ch01-data.x01-data-solutions :refer [data]]
[clojure.string :as cs]
[cuerdas.core :as cc])) | ||||||||||||||||||||||||||||||||||||||||
(comment
(->> data
(filter (fn [{:keys [powers]}] (get powers "Bullet Immunity")))
(mapcat :nemesis))
;; Find the alias of the nemesis of everyone with an alias.
(let [d (->> data (filter :alias) (mapcat :nemesis) (map :name) set)]
(->> data
(filter (fn [{:keys [name]}] (d name)))
(map :alias)))
) | ||||||||||||||||||||||||||||||||||||||||
Remove entries in a seq of pairs for which any of the following are true:
| (defn remove-bad-entries [m]
(into (empty m)
(remove (fn [[k v]] (or (nil? k) (contains? #{"-" nil} v))) m))) | |||||||||||||||||||||||||||||||||||||||
(comment
(remove-bad-entries
{nil "A" :a nil :b "" :c "OK"})) | ||||||||||||||||||||||||||||||||||||||||
Convert a sequence of vectors into a sequence of maps, assuming the first row of the vectors is a header row. | (defn table->maps [[headers & cols]]
(let [h (map cc/keyword headers)]
(->> cols
(map (fn [col] (zipmap h (map #(cond-> % (string? %) cs/trim) col))))
(map remove-bad-entries)))) | |||||||||||||||||||||||||||||||||||||||
Create the Primary Hero Data SetWe are going to parse the kaggle files found at the following links:
This exercise demonstrates the power of Clojure as a data-first language. Many other languages have libraries or DSLs for processing data, but Clojure is inherently a DSL for data. Data is generally loaded straight from a file, normalized into a desired format, and worked with directly. | (ns clojure-north-2020.ch01-data.x03-hero-data
(:require [clojure-csv.core :as csv]
[clojure-north-2020.ch01-data.x02-functions :refer
[csv-file->maps kwize maybe-bulk-update]])) | |||||||||||||||||||||||||||||||||||||||
Process heroes_information.csv to get basic superhero data | (defn normalize [m]
(let [dbl-fields [:height :weight]
kw-fields [:gender :alignment :hair-color :skin-color :eye-color :race]]
(-> m
(maybe-bulk-update dbl-fields #(Double/parseDouble %))
(maybe-bulk-update kw-fields kwize)))) | |||||||||||||||||||||||||||||||||||||||
(comment
(normalize
{:publisher "DC Comics"
:race "Martian"
:name "Martian Manhunter"
:alignment "good"
:weight "135.0"
:hair-color "No Hair"
:skin-color "green"
:eye-color "red"
:gender "Male"
:height "201.0"})) | ||||||||||||||||||||||||||||||||||||||||
(defn heroes-data []
(let [filename "resources/heroes_information.csv"]
(->> filename
csv-file->maps
(map normalize)))) | ||||||||||||||||||||||||||||||||||||||||
Exercise: Investigate Data QualityDetermine the following:
| (comment
(take 10 (heroes-data))
;Compute the frequency of names if the name occurs > 1 time.
(->> (heroes-data)
;...)
;Given a hero name, determine the duplicate values associated with
; non-distinct keys. You may want to use the test data from below.
(->> (heroes-data)
;...)
; Hint:
(do
(use 'clojure.repl)
(doc merge-with))
;TODO - Promote to the function "dupes"
(defn dupes [maps]
;Extract by promoting the previous exercise)
;We've build a generally useful function
(= {:age [12 14 16]
:height [100 101]}
(dupes
[{:name "Mark" :age 12}
{:name "Mark" :age 12 :height 100}
{:name "Mark" :age 14 :height 100}
{:name "Mark" :age 16 :height 101}]))) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x04-hero-data-solutions) | ||||||||||||||||||||||||||||||||||||||||
(comment
(require '[clojure-north-2020.ch01-data.x03-hero-data :refer [heroes-data]])
(take 10 (heroes-data))
;Compute the frequency of names if the name occurs > 1 time.
(->> (heroes-data)
(map :name)
frequencies
(filter (fn [[_ v]] (> v 1)))
(into {}))
;Given a hero name, determine the duplicate values associated with
; non-distinct keys
(->> (heroes-data)
(filter (comp #{"Spider-Man"} :name))
(apply merge-with (fn [a b] (if (= a b) a (flatten (vector a b)))))
(filter (fn [[_ v]] (seq? v)))
(into {}))
(defn dupes [m]
(->> m
(apply merge-with (fn [a b] (if (= a b) [a] (vector a b))))
(map (fn [[k v]] [k (distinct (flatten v))]))
(filter (fn [[_ [_ s]]] s))
(into {})))
(->> (heroes-data)
(filter (comp #{"Spider-Man"} :name))
dupes)
(dupes
[{:name "Mark" :age 12}
{:name "Mark" :age 12 :height 100}
{:name "Mark" :age 14 :height 100}])) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x04-hero-powers-data
(:require [clojure-csv.core :as csv]
[clojure-north-2020.ch01-data.x02-functions :refer [csv-file->maps table->maps]])) | ||||||||||||||||||||||||||||||||||||||||
Get Powers DataWe'll process super_hero_powers.csv to get powers data. Convert map of | (defn normalize [raw-hero-map]
(letfn [(f [m [k v]]
(cond
(= k :hero-names) (assoc m :name v)
(= v "True") (update m :powers (comp set conj) k)
:else m))]
(reduce f {} raw-hero-map))) | |||||||||||||||||||||||||||||||||||||||
Note that the dataset is a seq of maps with the key | (defn powers-data []
(let [filename "resources/super_hero_powers.csv"]
(->> filename
csv-file->maps
(map normalize)))) | |||||||||||||||||||||||||||||||||||||||
(comment
;Non-reduced powers
(let [filename "resources/super_hero_powers.csv"]
(->> filename
csv-file->maps
(take 2)))
; Reduced powers
(take 2 (powers-data))
; See who has the powers flight, super-strength, and vision-x-ray?
; Before trying it out, who might you expect to have these powers?
(->> (powers-data)
(filter (fn [{:keys [name powers]}]
(and (powers :flight)
(powers :super-strength)
(powers :vision-x-ray))))
(map :name))
) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x05-supplemental-hero-data
(:require [clojure-csv.core :as csv]
[clojure-north-2020.ch01-data.x02-functions :refer
[csv-file->maps kwize maybe-bulk-update maybe-update table->maps]]
[clojure.string :as cs])) | ||||||||||||||||||||||||||||||||||||||||
Supplemental Hero DataProcess SuperheroDataset.csv to get additional source data. This is a fairly ugly file and requires extra processing. We are not going to do any exercises here, but it is worthwhile to scroll to the bottom of the file and view the rich comments. | (defn remove-trash-fields [m]
(let [trash-values #{"No team connections added yet." "No alter egos found."}]
(into {} (remove (fn [[_ v]] (trash-values v)) m)))) | |||||||||||||||||||||||||||||||||||||||
(defn process-team-affiliations [s]
(let [teams (map cs/trim (cs/split s #","))
parser (partial re-matches #"(Formerly:)?([^\(]+)(?:\(([^\)]+)\))?")]
(loop [[team & r] teams former? false res []]
(if team
(let [[_ f n l] (parser team)
former? (or former? (some? f))
team-data {:team/name (cs/trim n)
:team/leader? (some? l)
:team/former? former?}]
(recur r former? (conj res team-data)))
res)))) | ||||||||||||||||||||||||||||||||||||||||
(defn captrim-all [v] (map (comp cs/capitalize cs/trim) v)) (defn process-occupations [s] (captrim-all (cs/split s #"[,;]"))) (defn process-bases [s] (captrim-all (cs/split s #";"))) (defn process-alter-egos [s] (captrim-all (cs/split s #","))) | ||||||||||||||||||||||||||||||||||||||||
(defn process-units-field [s]
(let [multipliers {"ton" 1000 "meter" 100}
[mag units] (-> s (cs/split #"//") last cs/trim (cs/split #"\s+"))]
(* (Double/parseDouble (cs/replace mag #"," ))
(multipliers units 1)))) | ||||||||||||||||||||||||||||||||||||||||
(defn process-aliases [s] (map cs/trim (cs/split s #","))) | ||||||||||||||||||||||||||||||||||||||||
(defn process-relatives [s]
(for [[_ names relation] (re-seq #";?([^;\(]*)\(([^\)]+)\)" s)
name (->> (cs/split names #",") (map cs/trim) (filter seq))]
{:relative {:name (cs/trim name)}
:relationship (kwize (cs/trim relation))})) | ||||||||||||||||||||||||||||||||||||||||
(defn gather-stats [m]
(let [attrs [:combat :durability :intelligence :power :speed :strength :total-power :unnamed-0]
stats (select-keys m attrs)]
(assoc
(apply dissoc m attrs)
:stats (map (fn [[k v]] {:stat/name k :stat/value v}) stats)))) | ||||||||||||||||||||||||||||||||||||||||
(defn normalize [m]
(let [dbl-fields [:speed :intelligence :unnamed-0 :power :durability :strength
:total-power :combat]
kw-fields [:alignment :hair-color :eye-color :gender :skin-color :race]
unit-fields [:weight :height]]
(-> m
(maybe-bulk-update dbl-fields #(Double/parseDouble %))
(maybe-bulk-update kw-fields kwize)
(maybe-bulk-update unit-fields process-units-field)
remove-trash-fields
(maybe-update :team-affiliation process-team-affiliations)
(maybe-update :aliases process-aliases)
(maybe-update :alter-egos process-alter-egos)
(maybe-update :occupation process-occupations)
(maybe-update :relatives process-relatives)
(maybe-update :base process-bases)
gather-stats))) | ||||||||||||||||||||||||||||||||||||||||
(defn supplemental-hero-data []
(let [filename "resources/SuperheroDataset.csv"]
(->> filename csv-file->maps (map normalize)))) | ||||||||||||||||||||||||||||||||||||||||
(comment
(take 10 (supplemental-hero-data))
(let [filename "resources/SuperheroDataset.csv"]
(->> filename
csv-file->maps
(map normalize)
(mapcat :relatives)
(map first)
(filter identity)
distinct))
) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch01-data.x06-yoda-quotes
(:require [clojure-csv.core :as csv]
[clojure-north-2020.ch01-data.x02-functions :refer [csv-file->maps]])) | ||||||||||||||||||||||||||||||||||||||||
Not used today :( Sorry! However, you can inspect it with our generic csv-file->maps function. | (comment (csv-file->maps "resources/yoda-corpus.csv")) | |||||||||||||||||||||||||||||||||||||||
A few utility functions for working with DatahikeNothing special to see here. | (ns clojure-north-2020.ch02-datalog.datahike-utils
(:require [clojure.edn :as edn]
[clojure.java.io :as io]
[datahike.api :as d])) | |||||||||||||||||||||||||||||||||||||||
Create a datahike connection backed by the given directory name. | (defn conn-from-dirname [dirname]
(let [db-dir (doto (io/file dirname) io/make-parents)
uri (str "datahike:" (io/as-url db-dir))
_ (when-not (d/database-exists? uri)
(d/create-database uri))
conn (d/connect uri)]
(alter-meta! conn assoc :uri uri)
conn)) | |||||||||||||||||||||||||||||||||||||||
Release a datahike connection and delete the database. | (defn cleanup [conn]
(let [{:keys [uri]} (meta conn)]
(d/release conn)
(when uri (d/delete-database uri)))) | |||||||||||||||||||||||||||||||||||||||
read an edn resource to data. | (defn read-edn [resource] (->> (io/resource resource) slurp edn/read-string)) | |||||||||||||||||||||||||||||||||||||||
(comment (conn-from-dirname "tmp/abc") ) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x00-basics (:require [datascript.core :as d])) | ||||||||||||||||||||||||||||||||||||||||
Datascript (and Datahike, Datomic, and Crux) are Fact Stores | (def db
(d/db-with
(d/empty-db)
[{:name "Mark" :favorite-food "Pizza"}
{:name "Pat" :favorite-food "Ice Cream" :age 42}
{:name "Chloe" :favorite-food "Chips"}])) | |||||||||||||||||||||||||||||||||||||||
"Facts" are DatomsEvaluate the above code in a REPL. | [[1 :favorite-food "Pizza" 536870913] [1 :name "Mark" 536870913] [2 :age 42 536870913] [2 :favorite-food "Ice Cream" 536870913] [2 :name "Pat" 536870913] [3 :favorite-food "Chips" 536870913] [3 :name "Chloe" 536870913]] | |||||||||||||||||||||||||||||||||||||||
These entries are
Datoms are often called "Facts" and the DBs themselves considered "Fact Stores." In plain terms, a fact is a statement made about something for some point in time and a datom describes such facts. Note that some implementations may also have additional fields such as a boolean indicating whether the fact was asserted or retracted. | ||||||||||||||||||||||||||||||||||||||||
QueriesQueries are done by unifying facts | (d/q
'[:find ?n
:in $
:where
[?e :name ?n]]
db) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x01-schemas
(:require [clojure-north-2020.ch02-datalog.datahike-utils :as du]
[datahike.api :as dh]
[datascript.core :as ds])) | ||||||||||||||||||||||||||||||||||||||||
SchemasA key concept in data modeling with datalog databases is that schema is specified at the attribute level.
Take a moment to consider this. This is a very, very powerful concept. | ||||||||||||||||||||||||||||||||||||||||
Schema-free DatascriptDatascript is very forgiving. You can pretty much dump whatever you want into a Datascript database. You get full query powers, but not all of your entities explode as expected and not all queries are as efficient. Execute the following in a REPL. How many entities are created? | (ds/db-with
(ds/empty-db)
[{:name "Batman"
:alias "Bruce Wayne"
:powers #{"Rich"}
:weapons #{"Utility Belt" "Kryptonite Spear"}
:hair-color :black
:alignment "Chaotic Good"
:nemesis [{:name "Joker"}
{:name "Penguin"}]}
;Try with and without this. What happens?
;{:name "Batman" :alias "Bruce"}
]) | |||||||||||||||||||||||||||||||||||||||
Datascript SchemasDatascript schemas are a map in which keys are the schema keys in the database and the values describe the keys. | (def datascript-schema
{:name {:db/unique :db.unique/identity}
:alias {:db/unique :db.unique/identity
:db/cardinality :db.cardinality/many}
:powers {:db/cardinality :db.cardinality/many}
:weapons {:db/cardinality :db.cardinality/many}
:nemesis {:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many}}) | |||||||||||||||||||||||||||||||||||||||
Datascript with SchemaNow evaluate the db with the full schema. | (ds/db-with
(ds/empty-db
datascript-schema)
[{:name "Batman"
:alias "Bruce Wayne"
:powers #{"Rich"}
:weapons #{"Utility Belt" "Kryptonite Spear"}
:hair-color :black
:alignment "Chaotic Good"
:nemesis [{:name "Joker"}
{:name "Penguin"}]}
{:name "Batman"
:alias "Bruce"}]) | |||||||||||||||||||||||||||||||||||||||
What happens when a key is missing? | (ds/db-with
(ds/empty-db
; Note that we're removing the alias cardinality/many attribute
(dissoc datascript-schema :alias))
[{:name "Batman"
:alias "Bruce Wayne"
:powers #{"Rich"}
:weapons #{"Utility Belt" "Kryptonite Spear"}
:hair-color :green
:alignment "Chaotic Good"
:nemesis [{:name "Joker"}
{:name "Penguin"}]}
; Observe that alias and hair color are overwritten
{:name "Batman"
:alias "Bruce"
:hair-color :black}]) | |||||||||||||||||||||||||||||||||||||||
Datahike/Datomic SchemasDatahike (and Datomic) schemas are a vector of maps in which each map describes a single attribute. Unlike Datascript, a schema must exist for every attribute to be transacted. | (def schema
[{:db/ident :name
:db/valueType :db.type/string
:db/unique :db.unique/identity
:db/cardinality :db.cardinality/one}
{:db/ident :alias
:db/valueType :db.type/string
:db/unique :db.unique/identity
:db/cardinality :db.cardinality/many}
{:db/ident :alignment
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one}
{:db/ident :powers
:db/valueType :db.type/string
:db/cardinality :db.cardinality/many}
{:db/ident :weapons
:db/valueType :db.type/string
:db/cardinality :db.cardinality/many}
{:db/ident :nemesis
:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many}]) | |||||||||||||||||||||||||||||||||||||||
(comment
(def conn (du/conn-from-dirname "tmp/x01-schemas"))
(dh/transact conn schema)
;What happens when we transact this? How do we fix it?
;Exercise - Fix the a schema.
(dh/transact
conn
[{:name "Batman"
:alias "Bruce Wayne"
:powers #{"Rich"}
:weapons #{"Utility Belt" "Kryptonite Spear"}
:hair-color :black
:alignment "Chaotic Good"
:nemesis [{:name "Joker"}
{:name "Penguin"}]}
])
(du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x02-queries
(:require [clojure-north-2020.ch02-datalog.x01-schemas :refer [datascript-schema]]
[datahike.api :as d]
[datascript.core :as ds])) | ||||||||||||||||||||||||||||||||||||||||
Queries and Data AccessIn this section we discuss the key ways to get data from a db. For Datascript and Datahike the APIs are the same except for time-travel functions. Other features are not identical across all implementations. For example, the entity API is not available in Datomic Cloud and as-of is not avaiable for Datascript. We'll just use Datascript for this example as the APIs are identical for what is being demonstrated. | (def dsdb
(ds/db-with
(ds/empty-db
datascript-schema)
[{:name "Batman"
:alias "Bruce Wayne"
:powers #{"Rich"}
:weapons #{"Utility Belt" "Kryptonite Spear"}
:hair-color :black
:alignment "Chaotic Good"
:nemesis [{:name "Joker"}
{:name "Penguin"}]}
{:name "Batman"
:alias "Bruce"}])) | |||||||||||||||||||||||||||||||||||||||
Exercise: Try the following in a REPL to understand the query methods. | ||||||||||||||||||||||||||||||||||||||||
The Pull APIThis allows you to 'pull' facts straight from a db given an identity. | (ds/pull dsdb '[*] 1) | |||||||||||||||||||||||||||||||||||||||
You can use 'lookup refs' for any unique identity (entity or value). | (ds/pull dsdb '[*] [:name "Batman"]) | |||||||||||||||||||||||||||||||||||||||
You can specify certain keys as well as do more complex attribute specs. | (ds/pull dsdb '[:name {:nemesis [:name]}] [:name "Batman"]) | |||||||||||||||||||||||||||||||||||||||
The Entity APIWhen you get an entity it's like situating yourself at a node in a graph db. | (:name (ds/entity dsdb [:name "Batman"])) | |||||||||||||||||||||||||||||||||||||||
You can navigate any way you want. This is a forward reference. | (->> (ds/entity dsdb [:name "Batman"])
:nemesis) | |||||||||||||||||||||||||||||||||||||||
You can also do 'backrefs' to walk the link backwards. | (->> (ds/entity dsdb [:name "Joker"])
:_nemesis) | |||||||||||||||||||||||||||||||||||||||
Exercise: How might you modify the last two queries to get the names of the nemeses? This is much more straightforward than our brute-force seq function driven approach. | ||||||||||||||||||||||||||||||||||||||||
The Query APIThis is the most powerful and most commonly used API. Queries have a powerful datalog syntax. Note that queries are data. It is common to inline them with the q function, but they can be stored up as standalone items in a file, db, etc. | (def nemeses-query
'[:find [?enemy-name ...]
:in $ ?name
:where
[?e :name ?name]
[?e :nemesis ?n]
[?n :name ?enemy-name]]) | |||||||||||||||||||||||||||||||||||||||
Get the nemeses of Batman | (ds/q nemeses-query dsdb "Batman") | |||||||||||||||||||||||||||||||||||||||
Exercise - Write a query to list the name and alignment of all individuals in the database. | (comment
(ds/q
'[:find ...]
;;Add some more facts to make it interesting
(ds/db-with
dsdb
[{:name "Joker" :alignment "Chaotic Evil"}
{:name "Darth Vader" :alignment "Lawful Evil"}]))) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x02-queries-solutions) | ||||||||||||||||||||||||||||||||||||||||
(comment
(require '[datascript.core :as ds])
(require '[clojure-north-2020.ch02-datalog.x02-queries :refer [dsdb]])
(ds/q
'[:find ?name ?alignment
:in $
:where
[?e :name ?name]
[?e :alignment ?alignment]]
(ds/db-with
dsdb
[{:name "Joker" :alignment "Chaotic Evil"}
{:name "Darth Vader" :alignment "Lawful Evil"}]))
) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x03-hero-schema (:require [clojure-north-2020.ch02-datalog.datahike-utils :as du])) | ||||||||||||||||||||||||||||||||||||||||
Loading the Hero Schema and Normalizing the DataHere we are loading in our pre-created basic hero schema. | (def schema (du/read-edn "schemas/datahike/hero-schema.edn")) | |||||||||||||||||||||||||||||||||||||||
It is common to have a function that transforms the data into a schema compliant format. This can also be achieved with database or transaction functions, which are beyond the scope of this workshop. | (defn hero->dh-format [{:keys [publisher] :as hero}]
(cond-> hero publisher (update :publisher (fn [p] {:name p})))) | |||||||||||||||||||||||||||||||||||||||
(comment
(require
'[clojure-north-2020.ch01-data.x03-hero-data :as hd]
'[datahike.api :as d])
(def conn (du/conn-from-dirname "tmp/hero-data-schema"))
(d/transact conn schema)
(count (d/transact conn (mapv hero->dh-format (hd/heroes-data))))
(d/pull @conn '[*] [:name "Spider-Man"])
(du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x04-hero-powers-schema (:require [clojure-north-2020.ch02-datalog.datahike-utils :as du])) | ||||||||||||||||||||||||||||||||||||||||
Loading the Hero Powers SchemaNo modification of data is required. | (def schema (du/read-edn "schemas/datahike/hero-powers-schema.edn")) | |||||||||||||||||||||||||||||||||||||||
(comment
(require
'[clojure-north-2020.ch01-data.x04-hero-powers-data :as hpd]
'[datahike.api :as d])
(def conn (du/conn-from-dirname "tmp/hero-powers-schema"))
(d/transact conn schema)
(count (d/transact conn (vec (hpd/powers-data))))
(d/pull @conn '[*] [:name "Spider-Man"])
;;NOTE! - The attribute is :powers, not :power!!!
;; ## Exercise: Who has the power :levitation?
(d/q
'[...]
@conn :levitation)
;; ## Exercise: Who has the same powers as the named super? Return a map of
;; power to sequence of names of hero with shared power.
(let [hero-name "Yoda"]
(->> (d/q
'[...]
@conn hero-name)
(reduce (fn [m [n p]] (update m p conj n)) {})))
;; ## Exercise: List heroes by number of powers. Who has the most?
(sort-by
second
(d/q
'[...]
@conn))
(du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x04-hero-powers-schema-solutions) | ||||||||||||||||||||||||||||||||||||||||
(comment
;; ## Exercise: Who has the power :levitation?
(d/q
'[:find [?name ...]
:in $ ?power
:where
[?e :name ?name]
[?e :powers ?power]]
@conn :levitation)
;; ## Exercise: Who has the same powers as the named super? Return a map of
;; power to sequence of names of hero with shared power.
(let [hero-name "Yoda"]
(->> (d/q
'[:find ?that-name ?power
:in $ ?name
:where
[?e :name ?name]
[?e :powers ?power]
[?f :powers ?power]
[?f :name ?that-name]
[(not= ?e ?f)]]
@conn hero-name)
(reduce (fn [m [n p]] (update m p conj n)) {})))
;; ## Exercise: List heroes by number of powers. Who has the most?
(sort-by
second
(d/q
'[:find ?name (count ?power)
:in $
:where
[?e :name ?name]
[?e :powers ?power]]
@conn))) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x05-supplemental-hero-data-schema (:require [clojure-north-2020.ch02-datalog.datahike-utils :as du])) | ||||||||||||||||||||||||||||||||||||||||
Loading the Supplemental Hero Schema and Normalizing the Data | (def schema (du/read-edn "schemas/datahike/supplemental-hero-data-schema.edn")) | |||||||||||||||||||||||||||||||||||||||
(defn add-stat-ids [{hero-name :name :as hero}]
(letfn [(add-stat-id [stats]
(map (fn [{stat-name :stat/name :as stat}]
(assoc stat :hero.stat/id (str hero-name "/" (name stat-name))))
stats))]
(update hero :stats add-stat-id))) | ||||||||||||||||||||||||||||||||||||||||
(defn add-relative-ids [{hero-name :name :as hero}]
(letfn [(add-relative-id [relatives]
(map (fn [{:keys [relative] :as r}]
(assoc r :hero.relative/id (str hero-name "/" (:name relative))))
relatives))]
(update hero :relatives add-relative-id))) | ||||||||||||||||||||||||||||||||||||||||
(defn add-team-affiliations [{hero-name :name :as hero}]
(letfn [(add-team-affiliation [team-affiliation]
(map (fn [{tn :team/name :as r}]
(assoc r :hero.team/id (str hero-name "/" tn)))
team-affiliation))]
(update hero :team-affiliation add-team-affiliation))) | ||||||||||||||||||||||||||||||||||||||||
(defn creator-name->creator-ref [{:keys [creator] :as hero}]
(cond-> hero creator (update :creator (fn [p] {:name p})))) | ||||||||||||||||||||||||||||||||||||||||
(defn hero->dh-format [hero]
(-> hero
add-stat-ids
add-relative-ids
add-team-affiliations
creator-name->creator-ref)) | ||||||||||||||||||||||||||||||||||||||||
(comment
(require
'[clojure-north-2020.ch01-data.x05-supplemental-hero-data :as shd]
'[datahike.api :as d])
(def conn (du/conn-from-dirname "tmp/supplemental-hero-data-schema"))
(d/transact conn schema)
(count (d/transact conn (mapv hero->dh-format (shd/supplemental-hero-data))))
;;Execute the following to see what data is provided by this dataset.
(d/pull @conn '[*] [:name "Spider-Man"])
(du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x06-the-ultimate-db
(:require [clojure-north-2020.ch01-data.x03-hero-data :as x03d]
[clojure-north-2020.ch01-data.x04-hero-powers-data :as x04d]
[clojure-north-2020.ch01-data.x05-supplemental-hero-data :as x05d]
[clojure-north-2020.ch02-datalog.datahike-utils :as du]
[clojure-north-2020.ch02-datalog.x03-hero-schema :as x03]
[clojure-north-2020.ch02-datalog.x04-hero-powers-schema :as x04]
[clojure-north-2020.ch02-datalog.x05-supplemental-hero-data-schema :as x05]
[datahike.api :as d])) | ||||||||||||||||||||||||||||||||||||||||
Create our Final SchemaNow that we have our 3 data sets with their corresponding schemas we can transact all of the data into one unified dataset. The schema can be combined into one as shown here or each can be transacted independently as seen in the comment block below. The truly powerful thing about this family of databases is the fact that no special logic, tables, schemas, etc. were needed to join the data into a unified data set. The existence of shared unique attributes provides implicit joins for a very interesting data set. | (def schema (vec (distinct (concat x03/schema x04/schema x05/schema)))) | |||||||||||||||||||||||||||||||||||||||
(comment (def conn (du/conn-from-dirname "tmp/the-ultimate-db")) (count @conn) (d/transact conn x03/schema) (d/transact conn x04/schema) (d/transact conn x05/schema) (keys (d/transact conn (mapv x03/hero->dh-format (x03d/heroes-data)))) (keys (d/transact conn (vec (x04d/powers-data)))) (keys (d/transact conn (mapv x05/hero->dh-format (x05d/supplemental-hero-data)))) (count @conn) ;;Once you load the above, try the following out. ;; ;; We now have a very cool set of information about our heroes, with ;; attributes such as teams, powers, occupations, stats, alter egos, and more. (d/pull @conn '[*] [:name "Spider-Man"]) ;; Note that this will destroy the db. (du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x07-queries (:require [datahike.api :as d])) | ||||||||||||||||||||||||||||||||||||||||
QueriesWe can now build a library of useful queries. Note that these are all data. You can store these as edn if desired, or maintain a ns or nses of queries. If care is taken, queries can be db-independent. All of these should work with both datascript and datahike. | ||||||||||||||||||||||||||||||||||||||||
find the name and alignment for all entities in the db. | (def alignment-query
'[:find ?name ?alignment
:in $
:where
[?e :name ?name]
[?e :alignment ?alignment]]) | |||||||||||||||||||||||||||||||||||||||
find the distinct alignments in db. | (def distinct-alignments-query
'[:find [?alignment ...]
:in $
:where
[_ :alignment ?alignment]]) | |||||||||||||||||||||||||||||||||||||||
find the nemeses of a given hero. | (def nemeses-query
'[:find [?nemesis-name ...]
:in $ ?name
:where
[?e :name ?name]
[?e :nemesis ?nemesis]
[?nemesis :name ?nemesis-name]]) | |||||||||||||||||||||||||||||||||||||||
Find powers shared between this super and others | (def shared-powers-query
'[:find ?that-name ?power
:in $ ?this-name
:where
[?e :name ?this-name]
[?e :powers ?power]
[?f :powers ?power]
[?f :name ?that-name]
[(not= ?e ?f)]]) | |||||||||||||||||||||||||||||||||||||||
Return all supers by name and power count | (def powers-by-count-query
'[:find ?name (count ?powers)
:in $
:where
[?e :name ?name]
[?e :powers ?powers]]) | |||||||||||||||||||||||||||||||||||||||
ExerciseReturn a set of 3-tuples (name, race, power) of all heroes in the db of the same race as the input hero. | (def shared-powers-by-race-query []) | |||||||||||||||||||||||||||||||||||||||
(def schema-query
'[:find [(pull ?e [*]) ...]
:in $
:where
[?e :db/ident ?ident]]) | ||||||||||||||||||||||||||||||||||||||||
(def name-query
'[:find [?name ...]
:in $
:where
[?e :name ?name]]) | ||||||||||||||||||||||||||||||||||||||||
Determine the set of valid values for each keyword (enum) type in the db. ExerciseWrite a query that determines all values for attributes where the type is schema. For example, what are the extant eye-colors or genders in the db? | (def distinct-ident-keywords-query []) | |||||||||||||||||||||||||||||||||||||||
(comment
(require '[clojure-north-2020.ch02-datalog.datahike-utils :as du]
'[datahike.api :as d]
'[clojure.set :refer [intersection]])
;;Use the previous "ultimate" db (don't clean it up yet).
(def conn (du/conn-from-dirname "tmp/the-ultimate-db"))
;;Reality check
(count @conn)
;; We now have a very cool set of information about our heroes, with
;; attributes such as teams, powers, occupations, stats, alter egos, and more.
(d/pull @conn '[*] [:name "Spider-Man"])
(d/pull @conn '[*] [:name "Odin"])
(d/pull @conn '[*] [:name "Thor"])
(d/pull @conn '[*] [:name "Spectre"])
(d/pull @conn '[*] [:name "Superman"])
(d/pull @conn '[*] [:name "Faora"])
(->> (d/q powers-by-count-query @conn)
(sort-by second))
;; ## Exercise
;; Determine the shared powers of heroes of a given race using only
;; their name. Using this query, determine the powers of Kryptonians.
(defn shared-powers-by-race [hero-name]
(->> (d/q
;Fill this in above. Might want to inline for the exercise.
shared-powers-by-race-query
@conn hero-name)
(group-by (fn [[n r]] [n r]))
(map (fn [[k v]]
(let [m (zipmap [:name :race] k)]
m (assoc m :powers (set (map last v))))))))
(shared-powers-by-race "Superman")
(shared-powers-by-race "Spider-Man")
(shared-powers-by-race "Thor")
;;Given the above result, determine all powers common to Kryptonians
;;and Asgardians by starting with an example of one.
;;Note - Data quality issue.
(:race (d/entity @conn [:name "Odin"]))
(:race (d/entity @conn [:name "Thor"]))
;; We can query the db using the schemas as well. The
;; distinct-ident-keywords-query should return the attribute-value
;; combinations for all idents in the db where the type is keyword.
(->> (d/q distinct-ident-keywords-query @conn)
(group-by first)
(map (fn [[k v]] [k (set (map second v))]))
(into {}))
(du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch02-datalog.x07-queries-solutions) | ||||||||||||||||||||||||||||||||||||||||
(def shared-powers-by-race-query
'[:find ?name ?race ?powers
:in $ ?n
:where
[?e :name ?n]
[?e :race ?race]
[?e :powers ?powers]
[?f :race ?race]
[?f :name ?name]
[?f :powers ?powers]
[(not= ?e ?f)]]) | ||||||||||||||||||||||||||||||||||||||||
Determine the set of valid values for each keyword (enum) type in the db. | (comment
;;All powers shared by Kryptonians
(->> "Superman"
shared-powers-by-race
(map :powers)
(apply intersection))
;;All powers shared by Asgardians
(->> "Thor"
shared-powers-by-race
(map :powers)
(apply intersection))
(def distinct-ident-keywords-query
'[:find ?ident ?v
:in $
:where
[?e :db/ident ?ident]
[?e :db/valueType :db.type/keyword]
[_ ?ident ?v]])) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x01-basic
(:require [clojure.pprint :as pp]
[ring.adapter.jetty :as jetty])) | ||||||||||||||||||||||||||||||||||||||||
Clojure Web App BasicsNow that we have a cool database, let's build a backend application that allows us to interact with it via RESTful service. We'll start with a simple explanation of what and how web apps work in Clojure. Once we understand how web apps work we'll wire in our database. Finally, we'll look at a technique for "pushing state to the edges" of our application for a truly functional app that still has state. The Basic Web AppHere we have the most basic Clojure application possible, consisting of exactly two things:
| ||||||||||||||||||||||||||||||||||||||||
(defn hello-handler [_request]
{:status 200
:body "Hello Clojure!"}) | ||||||||||||||||||||||||||||||||||||||||
(defn request-dump-handler
[request]
{:status 200
:body (with-out-str (pp/pprint request))}) | ||||||||||||||||||||||||||||||||||||||||
Exercise - Change the handler to request dump handler.This can be done as simply as evaluating (def handler request-dump-handler) in the REPL and refreshing your browser. | (def handler hello-handler) | |||||||||||||||||||||||||||||||||||||||
Our one and only web server. Notice that the hander is "var quoted" using
| (defonce server (jetty/run-jetty #'handler {:host "0.0.0.0"
:port 3000
:join? false})) | |||||||||||||||||||||||||||||||||||||||
(comment (require '[clojure.java.browse :refer [browse-url]]) (browse-url "http://localhost:3000") (.stop server)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x02-routes
(:require [clojure.pprint :as pp]
[ring.adapter.jetty :as jetty])) | ||||||||||||||||||||||||||||||||||||||||
Handler Roles/ConcernsHandlers have 3 main concerns:
| ||||||||||||||||||||||||||||||||||||||||
In this namespace we split out our API, handler, and routing logic. | ||||||||||||||||||||||||||||||||||||||||
Protip - Separate logic from handlersBusiness logic should know nothing about the calling context. If you are returning http response codes from or passing in web concepts to your business logic you are complecting your application. This particular "API" is contrived, but as we'll see in the future we can use completely independent API logic in our servers without the logic knowing anything about its surrounding context. | (defn greet [greetee] (format "Hello, %s!" (or greetee "Clojurian"))) | |||||||||||||||||||||||||||||||||||||||
Protip - Separate handlers from routingA well written handler contains very little, if any business logic. It should simply parse a request, invoke external business logic, and format a response, including setting proper response codes for non-happy-path execution. | (defn hello-handler [_request]
{:status 200
:body (greet nil)}) | |||||||||||||||||||||||||||||||||||||||
(defn request-dump-handler [request]
{:status 200
:body (with-out-str (pp/pprint request))}) | ||||||||||||||||||||||||||||||||||||||||
The Global HandlerHere we have some very simple routing logic based on case matching the uri from the request. Aside from demonstrating the concept that handlers are nothing more than a request->return map function you probably want to use a real routing library in real life like Reitit or Compojure. Protip - Only perform routing in the global handlerDefer handling of specific routes to their own functions. This facilitates independent testing of each handler and recomposition of routes. | (defn handler [{:keys [uri] :as request}]
(case uri
;; ### Exercise - modify the hello handler to greet the invoker by name if
;; the "name" query param is set. E.g. http://localhost:3000/hello?name=Mark
;; should return "Hello, Mark!" Tip - Use the request-dump-handler to
;; understand how to parse the request map.
"/hello" (hello-handler request)
"/dump" (request-dump-handler request)
{:status 404
:body "Sorry, I only understand hello and dump"})) | |||||||||||||||||||||||||||||||||||||||
(defonce server (jetty/run-jetty #'handler {:host "0.0.0.0"
:port 3000
:join? false})) | ||||||||||||||||||||||||||||||||||||||||
(comment (require '[clojure.java.browse :refer [browse-url]]) (browse-url "http://localhost:3000") (.stop server)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x02-routes-solutions) | ||||||||||||||||||||||||||||||||||||||||
(defn greet [greetee] (format "Hello, %s!" (or greetee "Clojurian"))) | ||||||||||||||||||||||||||||||||||||||||
(defn hello-handler [{:keys [query-string] :as _request}]
(let [[_ greetee] (some->> query-string (re-matches #"name=(.+)"))]
{:status 200
:body (greet greetee)})) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x03-responses
(:require [clojure.pprint :as pp]
[ring.adapter.jetty :as jetty]
[ring.util.http-response :refer [content-type header not-found ok]])) | ||||||||||||||||||||||||||||||||||||||||
Use ring.util.http-responseIn this namespace we introduce the ring.util.http-response library. It is added to your project with the [metosin/ring-http-response "0.9.1"] dependency. We can now formulate responses using simple functions instead of hand-rolling maps for each endpoint. Note that this library does nothing more than modify maps. | ||||||||||||||||||||||||||||||||||||||||
With the exception of response formatting, everything here is identical to the previous exercise. | ||||||||||||||||||||||||||||||||||||||||
Business Logic | (defn greet [greetee] (format "Hello, %s!" (or greetee "Clojurian"))) | |||||||||||||||||||||||||||||||||||||||
"Local" handlers | (defn hello-handler [{:keys [query-string] :as request}]
(let [[_ greetee] (some->> query-string (re-matches #"name=(.+)"))]
(ok (greet greetee)))) | |||||||||||||||||||||||||||||||||||||||
(defn request-dump-handler [request] (ok (with-out-str (pp/pprint request)))) | ||||||||||||||||||||||||||||||||||||||||
"Global" handler which is mostly routing to local handlers | (defn handler [{:keys [uri] :as request}]
(case uri
"/hello" (hello-handler request)
"/dump" (request-dump-handler request)
(not-found "Sorry, I don't understand that path."))) | |||||||||||||||||||||||||||||||||||||||
(defonce server (jetty/run-jetty #'handler {:host "0.0.0.0"
:port 3000
:join? false})) | ||||||||||||||||||||||||||||||||||||||||
ExerciseEvaluate the following forms to see that these helper functions are nothing more than simple data manipulators. | (comment
;Basic responses
(ok "All I do is modify maps.")
(not-found "You did't find me.")
;Use a threading macro with a basic response + response modifiers
(-> (ok "Watch this")
(content-type "text/plain")
(header "Content-Disposition" "attachment; filename=\"foo.txt\""))
(require '[clojure.java.browse :refer [browse-url]])
(browse-url "http://localhost:3000")
(.stop server)
) | |||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x04-reitit
(:require [clojure.pprint :as pp]
[muuntaja.core :as m]
[reitit.coercion.spec]
[reitit.ring :as ring]
[reitit.ring.coercion :as coercion]
[reitit.ring.middleware.exception :as exception]
[reitit.ring.middleware.multipart :as multipart]
[reitit.ring.middleware.muuntaja :as muuntaja]
[reitit.ring.middleware.parameters :as parameters]
[reitit.swagger :as swagger]
[reitit.swagger-ui :as swagger-ui]
[ring.adapter.jetty :as jetty]
[ring.util.http-response :refer [content-type header not-found ok]])) | ||||||||||||||||||||||||||||||||||||||||
Reitit: A library for routing + SwaggerThe details behind this ns are beyond the scope of this workshop, but the key detail to remember is that one of the concerns of a handler is routing. Until now we've just used a case statement on a request to do routing. This ns expands on that idea with the following additions:
| ||||||||||||||||||||||||||||||||||||||||
Business Logic | (defn greet [greetee] (format "Hello, %s!" (or greetee "Clojurian"))) | |||||||||||||||||||||||||||||||||||||||
"Local" handlers | (defn hello-handler [{:keys [params] :as request}]
(ok (greet (params "name")))) | |||||||||||||||||||||||||||||||||||||||
(defn request-dump-handler [request] (ok (with-out-str (pp/pprint request)))) | ||||||||||||||||||||||||||||||||||||||||
"Global" handler which is mostly routing to local handlers | (def router
(ring/router
[["/swagger.json"
{:get {:no-doc true
:swagger {:info {:title "my-api"}
:basePath "/"}
:handler (swagger/create-swagger-handler)}}]
["/basic"
{:swagger {:tags ["Basic Routes"]}}
["/hello"
{:get {:summary "Say hello"
:parameters {:query {:name string?}}
:responses {200 {:body string?}}
:handler hello-handler}}]
["/dump"
{:get {:summary "Dump the request"
:parameters {:query {:query-param string?}}
:responses {200 {:body string?}}
:handler request-dump-handler}}]]]
{:data {:coercion reitit.coercion.spec/coercion
:muuntaja m/instance
:middleware [parameters/parameters-middleware
muuntaja/format-negotiate-middleware
muuntaja/format-response-middleware
exception/exception-middleware
muuntaja/format-request-middleware
coercion/coerce-response-middleware
coercion/coerce-request-middleware
multipart/multipart-middleware]}})) | |||||||||||||||||||||||||||||||||||||||
(def handler
(ring/ring-handler
router
(ring/routes
(swagger-ui/create-swagger-ui-handler {:path "/"})
(ring/create-default-handler)))) | ||||||||||||||||||||||||||||||||||||||||
(defonce server (jetty/run-jetty #'handler {:host "0.0.0.0"
:port 3000
:join? false})) | ||||||||||||||||||||||||||||||||||||||||
(comment (require '[clojure.java.browse :refer [browse-url]]) (browse-url "http://localhost:3000") (.stop server) ) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x05-state
(:require [clojure-north-2020.ch01-data.x03-hero-data :as x03d]
[clojure-north-2020.ch01-data.x04-hero-powers-data :as x04d]
[clojure-north-2020.ch01-data.x05-supplemental-hero-data :as x05d]
[clojure-north-2020.ch02-datalog.datahike-utils :as du]
[clojure-north-2020.ch02-datalog.x03-hero-schema :as x03]
[clojure-north-2020.ch02-datalog.x04-hero-powers-schema :as x04]
[clojure-north-2020.ch02-datalog.x05-supplemental-hero-data-schema :as x05]
[clojure-north-2020.ch02-datalog.x07-queries :as x07]
[clojure.edn :as edn]
[clojure.pprint :as pp]
[datahike.api :as d]
[muuntaja.core :as m]
[reitit.coercion.spec]
[reitit.ring :as ring]
[reitit.ring.coercion :as coercion]
[reitit.ring.middleware.exception :as exception]
[reitit.ring.middleware.multipart :as multipart]
[reitit.ring.middleware.muuntaja :as muuntaja]
[reitit.ring.middleware.parameters :as parameters]
[reitit.swagger :as swagger]
[reitit.swagger-ui :as swagger-ui]
[ring.adapter.jetty :as jetty]
[ring.util.http-response :refer [bad-request not-found ok]])) | ||||||||||||||||||||||||||||||||||||||||
State: Let's make our app do somethingAny useful app will have some sort of backend state, usually a database. At this point we are finally going to pull in our really cool datahike db and add some cool queries and other interactions. | ||||||||||||||||||||||||||||||||||||||||
Here we create a global link to our ultimate db. This is actually terrible in practice, but is often done by inexperienced Clojurians as they aren't really sure how to handle stateful interactions as they try to develop functional apps. What we want to do is "push our state to the edges" as is often said by more advanced Clojurians. We'll discuss this concept in the next chapter and implement the concept. Despite the ugliness of how we're wiring in our db, it is a beautiful thing that we can arrive at this state from a completely bottom-up, data-driven design. We modeled our system as data, built a schema around it, created a few normalization functions as needed, and loaded our data into a database. Queries were data driven and reusable. At each stage of this process we built something useful and did not need to go back and modify our designs to wire in future stages of our system. With the exception of the db, everything until now has been purely functional. The db itself is a value when the current state is retrieved for use in queries. | (defonce conn (du/conn-from-dirname "tmp/the-ultimate-db")) | |||||||||||||||||||||||||||||||||||||||
Business Logic | (defn greet [greetee] (format "Hello, %s!" (or greetee "Clojurian"))) | |||||||||||||||||||||||||||||||||||||||
(defn load-schemas [] (d/transact conn x03/schema) (d/transact conn x04/schema) (d/transact conn x05/schema)) | ||||||||||||||||||||||||||||||||||||||||
(defn load-data []
(let [before (count @conn)
_ (d/transact conn (mapv x03/hero->dh-format (x03d/heroes-data)))
_ (d/transact conn (vec (x04d/powers-data)))
_ (d/transact conn (mapv x05/hero->dh-format (x05d/supplemental-hero-data)))
after (count @conn)]
{:datoms-before before
:datoms-after after
:datoms-added (- after before)})) | ||||||||||||||||||||||||||||||||||||||||
"Local" handlers | (defn hello-handler [{:keys [params] :as _request}]
(ok (greet (params "name")))) | |||||||||||||||||||||||||||||||||||||||
(defn request-dump-handler [request] (ok (with-out-str (pp/pprint request)))) | ||||||||||||||||||||||||||||||||||||||||
(defn load-schemas-handler [_request] (load-schemas) (ok "Schemas Loaded")) | ||||||||||||||||||||||||||||||||||||||||
(defn load-data-handler [_request] (ok (load-data))) | ||||||||||||||||||||||||||||||||||||||||
(defn hero-data-handler [{:keys [params] :as _request}]
(let [n (params "name")]
(try
(ok (d/pull @conn '[*] [:name n]))
(catch Throwable e
(not-found (format "Superhero \"%s\" not found." n)))))) | ||||||||||||||||||||||||||||||||||||||||
(defn datom-count-handler [_request]
(ok {:datoms (count @conn)})) | ||||||||||||||||||||||||||||||||||||||||
(defn schema-handler [_request]
(ok (map
#(dissoc % :db/id)
(d/q x07/schema-query @conn)))) | ||||||||||||||||||||||||||||||||||||||||
(defn hero-names-handler [_request] (ok (sort (d/q x07/name-query @conn)))) | ||||||||||||||||||||||||||||||||||||||||
(defn add-hero-handler [{:keys [body-params] :as _request}]
(try
(let [{:keys [tempids]} (d/transact conn [body-params])]
(ok tempids))
(catch Exception e
(bad-request (.getMessage e))))) | ||||||||||||||||||||||||||||||||||||||||
"Global" handler which is mostly routing to local handlers | (def router
(ring/router
[["/swagger.json"
{:get {:no-doc true
:swagger {:info {:title "my-api"}
:basePath "/"}
:handler (swagger/create-swagger-handler)}}]
["/query"
{:swagger {:tags ["Query Routes"]}}
["/load-schemas"
{:get {:summary "Load the schemas in the db."
:responses {200 {:body string?}}
:handler load-schemas-handler}}]
["/load-data"
{:get {:summary "Load the data in the db."
:responses {200 {:body {:datoms-before int?
:datoms-after int?
:datoms-added int?}}}
:handler load-data-handler}}]
["/hero"
{:get {:summary "Get data about a hero."
:parameters {:query {:name string?}}
:responses {200 {:body {}}
404 {:body string?}}
:handler hero-data-handler}}]
["/datom-count"
{:get {:summary "Get the number of datoms in the system."
:responses {200 {:body {:datoms int?}}}
:handler datom-count-handler}}]
["/schema"
{:get {:summary "Get the schema from the db."
:responses {200 {:body [{}]}}
:handler schema-handler}}]
["/names"
{:get {:summary "Get all superhero names"
:responses {200 {:body [string?]}}
:handler hero-names-handler}}]
["/add"
{:post {:summary "Add a new superhero"
:responses {200 {:body {}}}
:parameters {:body {:name string?}}
:handler add-hero-handler}}]
;; ## Exercise: Add an endpoint
]
["/basic"
{:swagger {:tags ["Basic Routes"]}}
["/hello"
{:get {:summary "Say hello"
:parameters {:query {:name string?}}
:responses {200 {:body string?}}
:handler hello-handler}}]
["/dump"
{:get {:summary "Dump the request"
:responses {200 {:body string?}}
:handler request-dump-handler}}]]]
{:data {:coercion reitit.coercion.spec/coercion
:muuntaja m/instance
:middleware [parameters/parameters-middleware
muuntaja/format-negotiate-middleware
muuntaja/format-response-middleware
exception/exception-middleware
muuntaja/format-request-middleware
coercion/coerce-response-middleware
coercion/coerce-request-middleware
multipart/multipart-middleware]}})) | |||||||||||||||||||||||||||||||||||||||
(def handler
(ring/ring-handler
router
(ring/routes
(swagger-ui/create-swagger-ui-handler {:path "/"})
(ring/create-default-handler)))) | ||||||||||||||||||||||||||||||||||||||||
(comment
(defonce server (jetty/run-jetty #'handler {:host "0.0.0.0"
:port 3000
:join? false}))
(require '[clojure.java.browse :refer [browse-url]])
(browse-url "http://localhost:3000")
(.stop server)
(du/cleanup conn)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch03-web.x05-state-solutions
(:require [clojure.edn :as edn]
[datahike.api :as d]
[ring.util.http-response :refer [bad-request ok]])) | ||||||||||||||||||||||||||||||||||||||||
(comment
(defn q-handler [{:keys [body-params] :as _request}]
(try
(let [q (edn/read-string (:query body-params))]
(ok (d/q q @conn)))
(catch Exception e
(bad-request (.getMessage e)))))
["/q"
{:post {:summary "Invoke a generic query on the db."
:parameters {:body {:query string?}}
:handler q-handler}}]
;Example request with content type of application/edn
{:query "[:find [(pull ?e [*]) ...]
:in $
:where
[?e :db/ident ?ident]]"}) | ||||||||||||||||||||||||||||||||||||||||
Don't look at this until we get to "x03-the-final-system." All will be revealed. | (ns clojure-north-2020.ch04-application.parts.datahike
(:require [clojure.java.io :as io]
[clojure.pprint :as pp]
[datahike.api :as d]
[integrant.core :as ig]
[taoensso.timbre :as timbre])) | |||||||||||||||||||||||||||||||||||||||
(defn file->datahike-db-uri [file]
(let [f (io/file file)
_ (io/make-parents f)]
(str "datahike:" (io/as-url f)))) | ||||||||||||||||||||||||||||||||||||||||
(defmethod ig/init-key ::database
[_ {:keys [db-uri db-file initial-tx schema-on-read temporal-index]
:as config}]
(if-some [uri (or db-uri (file->datahike-db-uri db-file))]
(let [args (cond-> [uri]
initial-tx (conj :initial-tx initial-tx)
schema-on-read (conj :schema-on-read schema-on-read)
temporal-index (conj :temporal-index temporal-index))]
(timbre/debug "Ensuring Datahike DB database.")
(when-not (d/database-exists? uri)
(timbre/debugf "Datahike DB database does not exist... Creating %s." uri)
(apply d/create-database args))
(timbre/debugf "Datahike DB uri: %s" uri)
(assoc config :db-uri uri))
(timbre/error "No uri provided for database"))) | ||||||||||||||||||||||||||||||||||||||||
(defmethod ig/halt-key! ::database [_ {:keys [db-uri delete-on-halt?]}]
(when delete-on-halt?
(timbre/debug "Deleting Datahike DB database.")
(d/delete-database db-uri))) | ||||||||||||||||||||||||||||||||||||||||
(defmethod ig/init-key ::connection [_ {:keys [db-uri db-config]}]
(if-some [uri (or db-uri (:db-uri db-config))]
(do
(timbre/debug "Creating Datahike DB connection.")
(d/connect uri))
(timbre/error "No db-uri provided for Datahike connection"))) | ||||||||||||||||||||||||||||||||||||||||
(defmethod ig/halt-key! ::connection [_ connection]
(if connection
(do
(timbre/debug "Releasing Datahike DB connection.")
(d/release connection))
(timbre/warn "No Datahike connection to release!"))) | ||||||||||||||||||||||||||||||||||||||||
Don't look at this until we get to "x03-the-final-system." All will be revealed. | (ns clojure-north-2020.ch04-application.parts.jetty
(:require [integrant.core :as ig]
[ring.adapter.jetty :as jetty]
[taoensso.timbre :as timbre])
(:import (org.eclipse.jetty.server Server))) | |||||||||||||||||||||||||||||||||||||||
This is a 'middleware' - a function that takes a handler and returns a new handler. For ring middlewares, both inbound handler and new hander take a request and return a response. | (defn wrap-component [handler component] (fn [request] (handler (into component request)))) | |||||||||||||||||||||||||||||||||||||||
(defmethod ig/init-key ::server [_ {:keys [handler] :as m}]
(timbre/debug "Launching Jetty web server.")
(jetty/run-jetty (wrap-component handler m) m)) | ||||||||||||||||||||||||||||||||||||||||
(defmethod ig/halt-key! ::server [_ ^Server server] (timbre/debug "Stopping Jetty web server.") (.stop server)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch04-application.x01-integrant
(:require [clojure-north-2020.ch03-web.x05-state :as x05-state]
[integrant.core :as ig]
[ring.adapter.jetty :as jetty]
[taoensso.timbre :as timbre])
(:import (org.eclipse.jetty.server Server))) | ||||||||||||||||||||||||||||||||||||||||
Building Reloadable Systems with IntegrantUntil now we've just used global, stateful components in our systems, namely a jetty web server and a datahike db as needed. This leaves lots of fragmented, global state sprinked about out program. There are also decoupled methods scattered about for initializing and cleaning up each component in the system. It's time to fix this by "pushing our state to the edges of our system." This means we want all stateful components in one controlled location that we can feed into one side of the system (the ingress edge) and treat as values at the other edge of the system (the egress side) such that the system appears functional to the user. Outside of the setup of the system (initialization) and the leaf functions/methods of the system where the actual components are used the entire system behaves as a collection of functions. One great aspect of our design so far is that we really are only a couple of steps away from achieving this design already. With the exception of our two stateful components (web server and connection) the entire api is nothing but functions and data. We don't need to adapt any of these building blocks to our new design. We'll just use our bottom-up set of functions as-is. Step 1: Make Centralized, Reloadable ComponentsIntegrant (aliased as ig) is a library for describing stateful resource configurations as maps. Upon initialization, each val in the config map is exchanged for a stateful thing using the config data in each key's value. The following example will illustrate the concept. Note that other good libraries exist for handling state, such as Component and Mount. | ||||||||||||||||||||||||||||||||||||||||
Web HandlersIn this ns we've just got our single OK handler. | (defn handler [_request]
{:status 200 :body "OK"}) | |||||||||||||||||||||||||||||||||||||||
Exercise - Change the handlerOnce you've started the system (below), swap out the handler for the one we've already developed. Note that we're decoupling yet another concern - system state management. You can just pull in existing functions and use them (However, this handler does have issues that we'll fix later). (def handler x05-state/handler) | ||||||||||||||||||||||||||||||||||||||||
ConfigurationHere is our config map. It has one key, ::server, which has configuration
options for a ring-jetty adapter. Recall that when ig/init is called the vals
of this map will be exchanged for a stateful component. In this case, we will
go from the map | (def config
{::server {:host "0.0.0.0"
:port 3000
:join? false
:handler #'handler}}) | |||||||||||||||||||||||||||||||||||||||
Integrant ImplementationTo use Integrant, implement the ig/init-key and ig/halt-key (optional) multimethods so that when the system is initialized logic is registered for each key. | (defmethod ig/init-key ::server [_ {:keys [handler] :as m}]
(timbre/debug "Launching Jetty web server.")
(jetty/run-jetty handler m)) | |||||||||||||||||||||||||||||||||||||||
(defmethod ig/halt-key! ::server [_ ^Server server] (timbre/debug "Stopping Jetty web server.") (.stop server)) | ||||||||||||||||||||||||||||||||||||||||
System BoilerplateWe'll now wrap the ig/init and ig/halt! methods in some standard boilerplate to create a restartable system with the following self-explanatory functions:
Exercise - Try them out.What we've done so far may not seem awesome, but we've done a few useful things:
| (defonce ^:dynamic *system* nil) | |||||||||||||||||||||||||||||||||||||||
(defn system [] *system*) | ||||||||||||||||||||||||||||||||||||||||
(defn start [] (alter-var-root #'*system* (fn [s] (if-not s (ig/init config) s)))) | ||||||||||||||||||||||||||||||||||||||||
(defn stop [] (alter-var-root #'*system* (fn [s] (when s (ig/halt! s) nil)))) | ||||||||||||||||||||||||||||||||||||||||
(defn restart [] (stop) (start)) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch04-application.x02-the-final-handler
(:require [clojure-north-2020.ch01-data.x03-hero-data :as x03d]
[clojure-north-2020.ch01-data.x04-hero-powers-data :as x04d]
[clojure-north-2020.ch01-data.x05-supplemental-hero-data :as x05d]
[clojure-north-2020.ch02-datalog.x03-hero-schema :as x03]
[clojure-north-2020.ch02-datalog.x04-hero-powers-schema :as x04]
[clojure-north-2020.ch02-datalog.x05-supplemental-hero-data-schema :as x05]
[clojure-north-2020.ch02-datalog.x07-queries :as x07]
[clojure.pprint :as pp]
[datahike.api :as d]
[muuntaja.core :as m]
[reitit.coercion.spec]
[reitit.ring :as ring]
[reitit.ring.coercion :as coercion]
[reitit.ring.middleware.exception :as exception]
[reitit.ring.middleware.multipart :as multipart]
[reitit.ring.middleware.muuntaja :as muuntaja]
[reitit.ring.middleware.parameters :as parameters]
[reitit.swagger :as swagger]
[reitit.swagger-ui :as swagger-ui]
[ring.util.http-response :refer [bad-request not-found ok]]
[clojure.edn :as edn])) | ||||||||||||||||||||||||||||||||||||||||
Threading State: A Purely Functional APIWe are now going to modify our API in a very simple way - We remove the global reference to the datahike connection and instead add it as a function parameter in our request. This may seem odd, but recall that Clojure handlers are nothing more than a function that takes a request and returns a response. All we need to do is inject our connection into the request to thread it through the handler. We'll do this in the next section. This approach has several advantages:
| ||||||||||||||||||||||||||||||||||||||||
Business Logic | (defn greet [greetee] (format "Hello, %s!" (or greetee "Clojurian"))) | |||||||||||||||||||||||||||||||||||||||
(defn load-schemas [conn] (d/transact conn x03/schema) (d/transact conn x04/schema) (d/transact conn x05/schema)) | ||||||||||||||||||||||||||||||||||||||||
(defn load-data [conn]
(let [before (count @conn)
_ (d/transact conn (mapv x03/hero->dh-format (x03d/heroes-data)))
_ (d/transact conn (vec (x04d/powers-data)))
_ (d/transact conn (mapv x05/hero->dh-format (x05d/supplemental-hero-data)))
after (count @conn)]
{:datoms-before before
:datoms-after after
:datoms-added (- after before)})) | ||||||||||||||||||||||||||||||||||||||||
"Local" handlers | (defn hello-handler [{:keys [params] :as _request}]
(ok (greet (params "name")))) | |||||||||||||||||||||||||||||||||||||||
(defn request-dump-handler [request] (ok (with-out-str (pp/pprint request)))) | ||||||||||||||||||||||||||||||||||||||||
Note that | (defn load-schemas-handler [{:keys [dh-conn] :as _request}]
(load-schemas dh-conn)
(ok "Schemas Loaded")) | |||||||||||||||||||||||||||||||||||||||
(defn load-data-handler [{:keys [dh-conn] :as _request}]
(ok (load-data dh-conn))) | ||||||||||||||||||||||||||||||||||||||||
(defn hero-data-handler [{:keys [params dh-conn] :as _request}]
(let [n (params "name")]
(try
(ok (d/pull @dh-conn '[*] [:name n]))
(catch Throwable e
(not-found (format "Superhero \"%s\" not found." n)))))) | ||||||||||||||||||||||||||||||||||||||||
(defn datom-count-handler [{:keys [dh-conn] :as _request}]
(ok {:datoms (count @dh-conn)})) | ||||||||||||||||||||||||||||||||||||||||
(defn schema-handler [{:keys [dh-conn] :as _request}]
(ok (map
#(dissoc % :db/id)
(d/q x07/schema-query @dh-conn)))) | ||||||||||||||||||||||||||||||||||||||||
(defn hero-names-handler [{:keys [dh-conn] :as _request}]
(ok (sort (d/q x07/name-query @dh-conn)))) | ||||||||||||||||||||||||||||||||||||||||
(defn add-hero-handler [{:keys [body-params dh-conn] :as _request}]
(try
(let [{:keys [tempids]} (d/transact dh-conn [body-params])]
(ok tempids))
(catch Exception e
(bad-request (.getMessage e))))) | ||||||||||||||||||||||||||||||||||||||||
(defn q-handler [{:keys [body-params dh-conn] :as _request}]
(try
(let [q (edn/read-string (:query body-params))]
(ok (d/q q @dh-conn)))
(catch Exception e
(bad-request (.getMessage e))))) | ||||||||||||||||||||||||||||||||||||||||
"Global" handler which is mostly routing to local handlers. Note that this DID NOT CHANGE AT ALL from the previous example. The state is at the edges, just being piped through. You could even put the hander/router in one ns, the function defs in another, and use an alias or import to swap out the handlers. | (def router
(ring/router
[["/swagger.json"
{:get {:no-doc true
:swagger {:info {:title "my-api"}
:basePath "/"}
:handler (swagger/create-swagger-handler)}}]
["/query"
{:swagger {:tags ["Query Routes"]}}
["/load-schemas"
{:get {:summary "Load the schemas in the db."
:responses {200 {:body string?}}
:handler load-schemas-handler}}]
["/load-data"
{:get {:summary "Load the data in the db."
:responses {200 {:body {:datoms-before int?
:datoms-after int?
:datoms-added int?}}}
:handler load-data-handler}}]
["/hero"
{:get {:summary "Get data about a hero."
:parameters {:query {:name string?}}
:responses {200 {:body {}}
404 {:body string?}}
:handler hero-data-handler}}]
["/datom-count"
{:get {:summary "Get the number of datoms in the system."
:responses {200 {:body {:datoms int?}}}
:handler datom-count-handler}}]
["/schema"
{:get {:summary "Get the schema from the db."
:responses {200 {:body [{}]}}
:handler schema-handler}}]
["/names"
{:get {:summary "Get all superhero names"
:responses {200 {:body [string?]}}
:handler hero-names-handler}}]
["/add"
{:post {:summary "Add a new superhero"
:responses {200 {:body {}}}
:parameters {:body {:name string?}}
:handler add-hero-handler}}]
;; ## Exercise: Add an endpoint
]
["/basic"
{:swagger {:tags ["Basic Routes"]}}
["/hello"
{:get {:summary "Say hello"
:parameters {:query {:name string?}}
:responses {200 {:body string?}}
:handler hello-handler}}]
["/dump"
{:get {:summary "Dump the request"
:responses {200 {:body string?}}
:handler request-dump-handler}}]]]
{:data {:coercion reitit.coercion.spec/coercion
:muuntaja m/instance
:middleware [parameters/parameters-middleware
muuntaja/format-negotiate-middleware
muuntaja/format-response-middleware
exception/exception-middleware
muuntaja/format-request-middleware
coercion/coerce-response-middleware
coercion/coerce-request-middleware
multipart/multipart-middleware]}})) | |||||||||||||||||||||||||||||||||||||||
(def handler
(ring/ring-handler
router
(ring/routes
(swagger-ui/create-swagger-ui-handler {:path "/"})
(ring/create-default-handler)))) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch04-application.x03-the-final-system
(:require [clojure-north-2020.ch02-datalog.x06-the-ultimate-db :as x06]
[clojure-north-2020.ch04-application.parts.datahike :as datahike]
[clojure-north-2020.ch04-application.parts.jetty :as jetty]
[clojure-north-2020.ch04-application.x02-the-final-handler :as tfh]
[integrant.core :as ig])) | ||||||||||||||||||||||||||||||||||||||||
Building a Library of Restartable ComponentsIn a prior section we saw how to create a restartable system that contains all of our stateful system components. These components are started and stopped via the init-key and halt-key! multimethods. We also saw how to create a handler that behaves as a function in which all stateful components (or even stateless components such as db as value) are injected into the handler function. Now it's time to combine these concepts to put everything together. Creating a library of ComponentsTo make things more modular and reusable, we are going to move our multimethods to their own namespace to create a library of components. Take a look at the following:
The wrap-component MiddlewareThe clojure-north-2020.ch04-application.parts.jetty ns now has this function, which wraps the inbound handler from the configuration map: ` (defn wrap-component [handler component] (fn [request] (handler (into component request)))) ` This Protip: Build a library of componentsWhether you use Integrant, Component, or Mount, you will likely find yourself using the same boilerplate code for a given library (e.g. Jetty or Datahike). Write a library to collect your implementations or use an existing one. Mine, called "Partsbin," can be found here. Configuring the ComponentsHere is our config map. It now has 3 components:
Recall that our new functional handler needs a dh-conn in each request to invoke datahike-aware functions. When ig/init is called the config map exchanges the values of the map for started instances of each value and the wrap-component middleware add the configured ::jetty/server keys to its own requests. This completes the threading of the stateful component through the handler. Another great advantage to this approach is that the handler knows NOTHING about the config, integrant, or anything else about its calling context. You can just as easily fabricate a request map and assoc a dh-conn into it and feed that into the handler. | (def config
{::jetty/server {:host "0.0.0.0"
:port 3000
:join? false
:handler #'tfh/handler
:dh-conn (ig/ref ::datahike/connection)}
::datahike/database {:db-file "tmp/the-ultimate-db"
:initial-tx x06/schema}
;::datahike/database {:db-file "tmp/the-final-dhdb"
; :delete-on-halt? true
; :initial-tx x06/schema}
::datahike/connection {:db-config (ig/ref ::datahike/database)}}) | |||||||||||||||||||||||||||||||||||||||
System BoilerplateSame as before | (defonce ^:dynamic *system* nil) | |||||||||||||||||||||||||||||||||||||||
(defn system [] *system*) | ||||||||||||||||||||||||||||||||||||||||
(defn start [] (alter-var-root #'*system* (fn [s] (if-not s (ig/init config) s)))) | ||||||||||||||||||||||||||||||||||||||||
(defn stop [] (alter-var-root #'*system* (fn [s] (when s (ig/halt! s) nil)))) | ||||||||||||||||||||||||||||||||||||||||
(defn restart [] (stop) (start)) | ||||||||||||||||||||||||||||||||||||||||
(comment
(keys (system))
(let [conn (::datahike/connection (system))]
(count @conn))
(use '[clojure.repl])
(require '[clojure.java.browse :refer [browse-url]])
(browse-url "http://localhost:3000")
) | ||||||||||||||||||||||||||||||||||||||||
(ns clojure-north-2020.ch04-application.x04-parting-thoughts) | ||||||||||||||||||||||||||||||||||||||||
In ConclusionIn this workshop, we've learned some key concepts that should help you in your Clojure journey from novice to full-stack, stateful system developer:
Additional MaterialIf you liked this workshop and want to learn more about any of the topics, feel free to check out these projects:
FeedbackI hope you enjoyed this workshop. I value any feedback you may have, positive or otherwise. Please feel free to reach our to me on twitter (@mark_bastian), gmail (markbastian@gmail.com), or linkedin. | ||||||||||||||||||||||||||||||||||||||||