Support Vector Machines

# Support Vector Machines
### Bradley C. Boehmke </br> and </br> Brandon M. Greenwell
### May 15, 2018

---

background-image: url(Images/i_svm.jpg)
background-position: center
background-size: contain

---
class: inverse, center, middle

# Introduction

---

## Support vector machines (SVMs)

* .large[A direct approach to .red[binary] classification]:

- Try to find a **hyperplane** in feature space that "best" separates the classes
    
* .large[In practice, it is difficult (if not impossible) to find a hyperplane to separate the classes in the original feature space]

* .large[SVMs extend this idea in two ways:]

- Relax/soften what we mean by separates
    
    - Use the [*kernel trick*](https://en.wikipedia.org/wiki/Kernel_method) to enlarge the feature space to the point that separation is (more) possible
    
]

]

---

## Hyperplanes

* A hyperplane in *p*-dimensional feature space is defined by the (linear) equation `$$f\left(X\right) = \beta_0 + \beta_1 X_1 + \dots + \beta_p X_p = 0$$`

- For `$p = 2$` the hyperplane is just a .red[line] in 2-D space
  
  - For `$p = 3$` the hyperplane is a .red[plane] in 3-D space
  
* `$f\left(X\right) > 0$` for points on one side of the hyperplane, and `$f\left(X\right) < 0$` for points on the other side
  
* It is (mathematically) convenient to code the binary outcome using {-1, 1} so that `$Y_i \times f\left(X_i\right) > 0$` for points on the correct side of the decision boundary!
    
]

---
class: inverse, center, middle

background-image: url(Images/separable.jpg)
background-position: center
background-size: contain

# Seperable data

---

## Separating hyperplanes

* For two separable classes, there are an infinite number of separating hyperplanes! 😱

* What we want is a separating hyperplane with .red[good generalization performance]!

* The .red[*hard margin classifier*] (HMC) is an "optimal" separating hyperplane and the simplest type of SVM

]

---
class: center, middle

## The .red[*optimal separating hyperplane*] separates the two classes .red[and] maximizes the distance to the closest point from either class

---

---

---
class: inverse, center, middle

background-image: url(Images/explain.png)
background-position: center
background-size: contain

---

---

---

---

---

---

---

---

## The hard margin classifier

<br/>

`$$\underset{\beta_0, \beta_1, \dots, \beta_p}{\text{maximize}} \quad M$$`

`$$\text{subject to:} \quad \begin{cases} \sum_{j = 1}^p \beta_j^2 = 1,\\ y_i\left(\beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip}\right) \ge M,\\ \quad \forall i = 1, 2, \dots, n \end{cases}$$`
]

<br/>

---
class: inverse, center, middle

# Sometimes perfect separation is achievable, but not desirable!

---

---

---

## The soft margin classifier

<br/>

`$$\underset{\beta_0, \beta_1, \dots, \beta_p}{\text{maximize}} \quad M$$`

`$$\text{subject to:} \quad \begin{cases} \sum_{j = 1}^p \beta_j^2 = 1,\\ y_i\left(\beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip}\right) \ge M\left(1 - \xi_i\right),\\ \quad \forall i = 1, 2, \dots, n\\ \xi_i \ge 0, \\ \sum_{i = 1}^n \xi_i \le C\end{cases}$$`
]

<br/>

---
class: center, middle

---
class: center, middle

## `$C$` is a regularization parameter

---
class: center, middle

<img src="GIFs/svmpath.gif" alt="drawing" style="width: 600px;"/>

---
class: center, middle

### As it turns out, you can fit the entire regularization path with essentially the same cost as a single SVM fit!

---
class: inverse, center, middle

background-image: url(Images/non-separable.jpg)
background-position: center
background-size: contain

# Non-separable data

---

## The support vector machine

* The objective function for the soft margin classifier can be re-expressed as

`$$L_D = \sum_{i = 1}^n - \frac{1}{2}\sum_{i = 1}^n\sum_{i' = 1}^n \alpha_i\alpha_{i'}y_iy_{i'}x_i^{\top}x_{i'}$$`

]

* In the SVM, we replace the .red[*dot product*] with a .red[*kernel function*]

`$$L_D = \sum_{i = 1}^n - \frac{1}{2}\sum_{i = 1}^n\sum_{i' = 1}^n \alpha_i\alpha_{i'}y_iy_{i'}K\left(x_i, x_{i'}\right)$$`

]

---
class: center, middle

.larger[Replacing the dot product with a kernel function is similar in spirit to .red[enlarging the feature space using basis functions] (e.g., like in MARS!)]

---

## Popular kernel functions

* .magenta[*d*-th degree polynomial:] `$$K\left(x, x'\right) = \left(1 + \langle x, x' \rangle\right) ^ d$$`

* .magenta[Radial basis function:] `$$K\left(x, x'\right) = \exp\left(\gamma \lVert x - x'\rVert ^ 2\right)$$`

* .magenta[Hyperbolic tangent:] `$$K\left(x, x'\right) = \tanh\left(k_1\lVert x - x'\rVert + k_2\right)$$`

]

---
class: center, middle

---
class: center, middle

Enlarge the feature space by adding a third variable

`$$X_3 = X_1^2 + X_2^2$$`

]

---
class: center, middle

<div id="htmlwidget-f4cd82c9c808c2aac388" style="width:504px;height:504px;" class="plotly html-widget"></div>
<script type="application/json" data-for="htmlwidget-f4cd82c9c808c2aac388">{"x":{"visdat":{"3bdd7bee3e9":["function () ","plotlyVisDat"]},"cur_data":"3bdd7bee3e9","attrs":{"3bdd7bee3e9":{"x":{},"y":{},"z":{},"color":{},"colors":["#1B9E77","#D95F02"],"alpha_stroke":1,"sizes":[10,100],"spans":[1,20]}},"layout":{"margin":{"b":40,"l":60,"t":25,"r":10},"scene":{"xaxis":{"title":"x1"},"yaxis":{"title":"x2"},"zaxis":{"title":"x3"}},"hovermode":"closest","showlegend":true},"source":"A","config":{"modeBarButtonsToAdd":[{"name":"Collaborate","icon":{"width":1000,"ascent":500,"descent":-50,"path":"M487 375c7-10 9-23 5-36l-79-259c-3-12-11-23-22-31-11-8-22-12-35-12l-263 0c-15 0-29 5-43 15-13 10-23 23-28 37-5 13-5 25-1 37 0 0 0 3 1 7 1 5 1 8 1 11 0 2 0 4-1 6 0 3-1 5-1 6 1 2 2 4 3 6 1 2 2 4 4 6 2 3 4 5 5 7 5 7 9 16 13 26 4 10 7 19 9 26 0 2 0 5 0 9-1 4-1 6 0 8 0 2 2 5 4 8 3 3 5 5 5 7 4 6 8 15 12 26 4 11 7 19 7 26 1 1 0 4 0 9-1 4-1 7 0 8 1 2 3 5 6 8 4 4 6 6 6 7 4 5 8 13 13 24 4 11 7 20 7 28 1 1 0 4 0 7-1 3-1 6-1 7 0 2 1 4 3 6 1 1 3 4 5 6 2 3 3 5 5 6 1 2 3 5 4 9 2 3 3 7 5 10 1 3 2 6 4 10 2 4 4 7 6 9 2 3 4 5 7 7 3 2 7 3 11 3 3 0 8 0 13-1l0-1c7 2 12 2 14 2l218 0c14 0 25-5 32-16 8-10 10-23 6-37l-79-259c-7-22-13-37-20-43-7-7-19-10-37-10l-248 0c-5 0-9-2-11-5-2-3-2-7 0-12 4-13 18-20 41-20l264 0c5 0 10 2 16 5 5 3 8 6 10 11l85 282c2 5 2 10 2 17 7-3 13-7 17-13z m-304 0c-1-3-1-5 0-7 1-1 3-2 6-2l174 0c2 0 4 1 7 2 2 2 4 4 5 7l6 18c0 3 0 5-1 7-1 1-3 2-6 2l-173 0c-3 0-5-1-8-2-2-2-4-4-4-7z m-24-73c-1-3-1-5 0-7 2-2 3-2 6-2l174 0c2 0 5 0 7 2 3 2 4 4 5 7l6 18c1 2 0 5-1 6-1 2-3 3-5 3l-174 0c-3 0-5-1-7-3-3-1-4-4-5-6z"},"click":"function(gd) { \n        // is this being viewed in RStudio?\n        if (location.search == '?viewer_pane=1') {\n          alert('To learn about plotly for collaboration, visit:\\n https://cpsievert.github.io/plotly_book/plot-ly-for-collaboration.html');\n        } else {\n          window.open('https://cpsievert.github.io/plotly_book/plot-ly-for-collaboration.html', '_blank');\n        }\n      }"}],"cloud":false},"data":[{"x":[0.184880691114813,-0.173724585678428,0.292430663481355,-0.353487743064761,0.0315697393380105,-0.339099933858961,0.41848012432456,-0.527459796052426,-0.749379522632807,-0.358715728390962,-0.149715784937143,0.152425371576101,0.639098391868174,0.0164024936966598,-0.597904153633863,-0.00299446796998382,0.343608122318983,0.155500024557114,0.381532086059451,-0.218610550276935,-0.19800116866827,-0.478924263268709,0.535735310520977,0.592049826402217,-0.385980625636876,0.173376116901636,-0.430689551867545,0.00336658209562302,0.00445355381816626,-0.238008493091911,0.0332018593326211,0.147532635834068,0.209851063787937,-0.231356921605766,0.428907586727291,0.338241091929376,-0.0645638755522668,-0.627942373976111,0.50433291727677,0.656048898585141,0.193973562214524,-0.489999345038086,0.274179557804018,0.154506682418287,-0.298041669186205,0.423718914855272,0.0243955790065229,0.416591171175241,-0.176168021280318,-0.425815349910408,-0.172342660371214,-0.170215421821922,-0.113058666232973,0.375482445582747,0.124500084202737,0.0796845499426126,-0.292996355798095,0.182135016657412,-0.32973230117932,0.416741462424397,-0.0176928448490798,-0.502019797451794,0.38006243435666,0.167331502307206,-0.377122760284692,-0.705128489527851,0.0920147793367505,0.170625266153365,-0.402562960051,0.539085494354367,0.128605844918638,-0.510736586991698,0.528063585981727,0.197031053248793,0.0316666699945927,-0.274231788702309,0.586885037366301,-0.258564332034439,0.419292179401964,-0.34251793473959,0.485480901319534,-0.417591316625476,-0.494222304783762,0.362606673501432,-0.250122302677482,-0.10235261451453],"y":[-0.328389044385403,-0.554052996449172,-0.57710592681542,-0.0629461016505957,0.739656292833388,-0.178174387197942,0.57638765592128,-0.153838477097452,-0.0223371940664947,-0.344745384063572,-0.228060669731349,-0.223822586238384,0.00481663085520267,0.766577545553446,0.336330285761505,-0.260779718868434,-0.587533293757588,0.0937672033905983,-0.517937834840268,-0.152486606966704,-0.271946588531137,-0.21221406199038,-0.570170728955418,0.285612549167126,0.00865495344623923,0.591875788290054,-0.239123679231852,-0.619909950066358,-0.155810560099781,0.000341253355145454,-0.318451403174549,0.48657774599269,0.647900882642716,-0.709922052454203,0.374871363397688,-0.0033267792314291,0.716216979082674,-0.381631572730839,0.24695616774261,-0.272541335318238,0.0521658444777131,-0.413892921991646,0.394172464963049,0.272079286165535,0.0384179367683828,-0.527444468345493,0.241506533697248,0.658341867849231,-0.488617597147822,0.664787280838937,-0.728723065461963,-0.403299115598202,-0.738116944208741,-0.199582407251,0.26649435935542,-0.0118553130887449,-0.0725201163440943,0.184052158147097,-0.400454944465309,-0.0289806257933378,-0.157790115103126,-0.538068289868534,0.700510294176638,0.766819896176457,-0.0609740521758795,0.150238844566047,0.244876070413738,-0.322758095338941,0.0925378319807351,0.562073536682874,-0.353344581555575,-0.524661109782755,-0.162025086116046,-0.364461571909487,0.596164704300463,0.143882780335844,-0.372372302226722,-0.0032848846167326,-0.605405771173537,0.686898790765554,-0.226093098521233,-0.284762403462082,-0.0844957921653986,0.292257664259523,-0.21132322633639,0.734159491490573],"z":[0.142020234419449,0.337154954543448,0.418566943709631,0.128915796210026,0.5480880799699,0.146734877396514,0.507348344353128,0.30188011348733,0.562068619180139,0.247526353628198,0.0744264853376563,0.0733300440105513,0.408469954421282,0.587910175146214,0.470607438053042,0.068015028611523,0.463261912997217,0.0329725460689568,0.413826333451901,0.0710427379966025,0.113159409807724,0.274403258053933,0.61210698309656,0.43209752518464,0.149055951586191,0.380376226675862,0.242673624057244,0.384299680066281,0.0242967647802186,0.0566481592377347,0.102513659646983,0.258523781531475,0.46381302270214,0.557515345735685,0.32449025704787,0.114418103729631,0.51713525515264,0.539954282339768,0.315339040235013,0.504678936791824,0.0403470181682646,0.411406709012536,0.230546362052653,0.0978994528722527,0.0903047744368385,0.457735385994389,0.0589205500935232,0.606962218864373,0.269782327964334,0.62326084098455,0.560739298720109,0.191623466468307,0.557598885338429,0.180820204224905,0.0865195145347442,0.00649017594798889,0.0911060317855251,0.0670483612113996,0.269087552987724,0.174513323174999,0.0252107571831112,0.541541361595589,0.63516212625655,0.616012584836459,0.145939411363492,0.51977789716037,0.0684310094776651,0.133285769556727,0.170620187152718,0.606539830862487,0.141391856661921,0.5361211414104,0.305103279370792,0.171653473343063,0.356415132642205,0.0959053284120405,0.483095178550074,0.0668663042673608,0.542322079477873,0.589148484373454,0.286809794744958,0.2554721341465,0.251395225439431,0.216898141986206,0.107218672285907,0.549466216643656],"type":"scatter3d","mode":"markers","name":"1","marker":{"color":"rgba(27,158,119,1)","line":{"color":"rgba(27,158,119,1)"}},"textfont":{"color":"rgba(27,158,119,1)"},"error_y":{"color":"rgba(27,158,119,1)"},"error_x":{"color":"rgba(27,158,119,1)"},"line":{"color":"rgba(27,158,119,1)"},"frame":null},{"x":[-0.826535359956324,-0.375871799420565,-0.385314610321075,-0.903558305930346,-0.931924305856228,0.940356121398509,-0.866437956225127,0.251865916419774,0.938579450361431,-0.313154518138617,0.827255330048501,-0.99755409732461,0.980633036699146,-0.390574976801872,0.882167169358581,0.241879029665142,0.98596209147945,-0.496926855761558,-0.802469267509878,-0.92185795865953,-0.313461131881922,0.541716917417943,0.48475546669215,0.842027988750488,0.490718647371978,0.662456168793142,-0.758111061528325,0.821340544614941,0.839587319642305,-0.974398004356772,-0.0979393678717315,-0.413724592886865,-0.0172268911264837,-0.935985750984401,0.635913654696196,-0.721876837778836,0.681055420543998,-0.541716842912138,0.850459226872772,-0.922854135744274,-0.997988515999168,-0.899095716886222,-0.338922295253724,-0.878189291339368,-0.324567660223693,0.977779885288328,-0.855937000829726,-0.183485923334956,0.123122045770288,0.7816789210774,-0.607691532932222,-0.791732123121619,0.0468979519791901,0.914261479396373,-0.901053602807224,-0.0949012534692883,-0.656630616169423,0.935521764215082,0.935268397442997,0.766513522714376,0.658995151519775,0.932834644336253,-0.324357732199132,0.99739996297285,-0.735172631684691,0.948652467224747,0.887322943657637,0.116164406761527,0.798050140962005,-0.821682870388031,0.0246450542472303,-0.287558771669865,-0.491487877443433,-0.601296406704932,0.0086163030937314,-0.760322855785489,-0.814160305541009,-0.818708814680576,-0.482826304156333,-0.686280146706849,0.107927126809955,-0.219207767397165,0.741550719365478,0.846395653206855,0.897573848254979,-0.815558804664761,-0.893078964203596,0.483015685342252,0.286179932300001,0.911439069546759,0.676391781307757,-0.934772125910968,-0.502985518891364,0.584045792929828,-0.961835874710232,-0.337054906412959,0.483049250207841,-0.12089617876336,0.869917268399149,-0.230735914316028,-0.587768248282373,-0.530938375275582,-0.46020361687988,0.932190658990294,0.622648072429001,-0.985263374634087,-0.812265664804727,-0.0340475980192423,0.929374325554818,0.778682630043477,-0.903905731160194,0.629260713700205,0.0764157236553729,0.61100801313296],"y":[-0.448570422362536,-0.952501417603344,0.804793477524072,0.960105892736465,0.197862752713263,0.370068506803364,-0.460782914888114,-0.894731002859771,0.900913139805198,0.904125681146979,0.209950249642134,-0.758683393709362,-0.203591211698949,-0.723860743455589,0.583883677143604,-0.910216906573623,0.834435044787824,0.717888657003641,0.788971993140876,-0.968182685319334,0.986558234784752,-0.76740355649963,0.761008152738214,0.214116952847689,0.646810280159116,0.847574689425528,0.615453876089305,0.485583499539644,0.231505441945046,-0.680569640826434,-0.948976122774184,-0.698345808777958,0.841660569421947,-0.295524003449827,-0.849677632562816,0.968234044034034,-0.628695658408105,0.593366112560034,-0.61134425457567,-0.796020167414099,0.934094183612615,0.924217897932976,-0.919339095242321,-0.925928303971887,-0.822546544019133,0.199783223215491,-0.864960380829871,-0.855476054362953,-0.89787164516747,0.602090126369148,0.658385285641998,0.761620802804828,0.808541218750179,0.125890491064638,0.991882580332458,0.845719256438315,0.985925229731947,-0.400757353752851,0.630039448849857,-0.781988210976124,0.800566408783197,0.707076358608902,0.772710749413818,-0.690173622220755,-0.928352592047304,0.571687162388116,-0.840080255176872,-0.921853886451572,0.932619445025921,-0.963551516178995,0.995406137779355,-0.85256107384339,0.781894484069198,0.985052986536175,-0.986590536311269,0.329629204701632,-0.508131122682244,0.665389012079686,-0.831574554089457,-0.49144511250779,0.961714936420321,-0.960252406075597,0.675686817616224,-0.417085194494575,-0.141536862589419,0.0237136026844382,0.330493747722358,-0.79609045246616,-0.959666574373841,-0.739428359549493,-0.729339069221169,-0.901880093384534,-0.946322016883641,-0.756261762231588,0.759832636453211,-0.96976669318974,-0.882027635350823,-0.837283190339804,0.808633930049837,0.900724616833031,0.839467057026923,-0.620205798186362,0.725803882814944,0.265484168659896,0.997449893970042,0.0282089752145112,0.555173575412482,0.941088281571865,-0.0428532613441348,0.217072224710137,-0.891117743216455,-0.740939968731254,0.930796172004193,0.588420444633812],"z":[0.884376125076634,1.04853856013603,0.79615989039217,1.738220937483,0.907632580756882,1.02122033477932,0.963035626640362,0.863980007332425,1.69257587011443,0.915508999540119,0.728430488418449,1.57071466897947,1.00309053414684,0.676523188419861,1.11913906312887,0.887000282004207,1.66840308980459,0.76230042383156,1.26643373125853,1.78719980809607,1.0715550318223,0.88236543714508,0.814121271021952,0.754857203335971,0.659168329398097,1.15723102972687,0.953515855204953,0.910391625253535,0.758501636954322,1.41262650690908,0.910147801374637,0.658854907397143,0.70868927989756,0.963403762660846,1.12633825550663,1.45858333294807,0.859094716753564,0.6455404814293,1.09702269417571,1.48530786279028,1.86851302192514,1.6625518309828,0.960052694261023,1.6285596555194,0.781926983140902,0.995966840352826,1.48078460979474,0.765506363650487,0.8213325293504,0.973534455927946,0.802760183547409,1.20690600204774,0.655938320317867,0.85172246844854,1.79572864829885,0.724247308620618,1.40321232471303,1.03580742790709,1.27167668236262,1.19904854260964,1.0751811845986,1.37013745057757,0.702289840697028,1.47114631494756,1.40231733353914,1.22676771521098,1.49307684147929,0.863308757364126,1.50666305672993,1.60359426381995,0.991440757827661,0.809550431797275,0.852919317891689,1.3318867550001,0.97343512701796,0.686746257622032,0.921054240956946,1.11302666063205,0.924637478994343,0.712498738371769,0.936543883635395,0.970136728661223,1.0064501448918,0.890345661234036,0.825671496542908,0.665698498818489,0.906816153586537,0.867064160794421,1.00285888762166,1.37747547640232,0.989441319713081,1.68718663022341,1.14851979185314,0.913041341252659,1.50247388529904,1.05405344915721,1.01130932764892,0.715659026865182,1.41064488668688,0.864543897524264,1.05017645352237,0.666550790444257,0.738578645298573,0.939461268517794,1.3825969130804,0.971539663678002,0.967993209056944,0.886806392642766,0.865573039008301,0.653466989072034,1.61113640309943,0.944961083069964,0.872220876639434,0.71956941177574],"type":"scatter3d","mode":"markers","name":"2","marker":{"color":"rgba(217,95,2,1)","line":{"color":"rgba(217,95,2,1)"}},"textfont":{"color":"rgba(217,95,2,1)"},"error_y":{"color":"rgba(217,95,2,1)"},"error_x":{"color":"rgba(217,95,2,1)"},"line":{"color":"rgba(217,95,2,1)"},"frame":null}],"highlight":{"on":"plotly_click","persistent":false,"dynamic":false,"selectize":false,"opacityDim":0.2,"selected":{"opacity":1},"debounce":0},"base_url":"https://plot.ly"},"evals":["config.modeBarButtonsToAdd.0.click"],"jsHooks":[]}</script>

---
class: center, middle

## The kernel trick

---

---

---

## Support vector machines

#### .green[Advantages:]

* Maximization of generalizability

* Global optimum (.green[convex optimization problem])

* Robustness to outliers

* Incredibly flexible

]

#### .red[Disadvantages:]

* Extension to more than two classes

* Doesn't scale well to large `$N$` 😞

* Predicting .red[class probabilities] is not automatic

]

---

## More than two classes

* The SVM, as introduced, is .red[applicable to only two classes!]

* What do we do when we have more than two classes?

- .orange.bold[*One-versus-all* (OVA):] Fit an SVM for each class (one class versus the rest); classify to the class with the largest; classify to the class for which the margin is the largest
  
  - .orange.bold[*One-versus-one* (OVO):] Fit all `$\binom{\# \ classes}{2}$` pairwise SVMs; classify to the class that wins the most pairwise competitions

]

---

## Support vector regression

---
class: inverse, center, middle

# Fitting SVMs in R

---
class: center, middle

---

## Lots of R packages available!!!

* [`e1071`](https://cran.r-project.org/package=e1071)

- Provides an interface to the award-winning `libsvm` C++, a very efficient implementations of SVMs
    
* [`kernlab`](https://cran.r-project.org/package=kernlab)

- More flexible implementation of SVMs with basic kernel functionality (**kern**al **lab**oratory), but still makes use of `libsvm` optimizers

* [`svmpath`](https://cran.r-project.org/package=svmpath)

- Computes the entire regularization path for the two-class SVM classifier with essentially the same cost as a single SVM fit!

* [`LiblineaR`](https://cran.r-project.org/package=LiblineaR)

- A wrapper around the `LIBLINEAR` C/C++ library for machine learning

---

## Two spirals benchmark problem

```r
# Load required packages
library(kernlab)  # for fitting SVMs
*library(mlbench)  # for ML benchmark data sets

# Simulate train and test sets
set.seed(0841)
trn <- as.data.frame(
  mlbench.spirals(300, cycles = 2, sd = 0.09)
)
tst <- as.data.frame(
  mlbench.spirals(10000, cycles = 2, sd = 0.09)
)
names(trn) <- names(tst) <- c("x1", "x2", "classes")
```

---

---

## Two spirals benchmark problem

```r
# Fit an SVM using a radial basis function kernel
spirals_rbf <- ksvm(
  classes ~ x1 + x2, data = trn, 
* kernel = "rbfdot",
* C = 500,  # I just picked a value
* prob.model = TRUE
)
```

---

---

## Two spirals benchmark problem

```r
# Test set confusion matrix
(tab <- table(
  pred = predict(spirals_rbf, newdata = tst),  # predicted outcome
  obs = tst$classes                    # observed outcome
))
```

```
##     obs
## pred    1    2
##    1 4580  424
##    2  420 4576
```

```r
# Test set error
1 - sum(diag(tab)) / nrow(tst)
```

```
## [1] 0.0844
```

---
class: center, middle, inverse

# Model tuning

???

Image credit: [imgflip](http://www.learntoplaymusic.com/blog/tune-guitar/)

---

## Using caret::getModelInfo()

```r
# Linear (i.e., ordinary inner product)
caret::getModelInfo("svmLinear")$svmLinear$parameters
```

```
##   parameter   class label
## 1         C numeric  Cost
```

```r
# Polynomial kernel
caret::getModelInfo("svmPoly")$svmPoly$parameters
```

```
##   parameter   class             label
## 1    degree numeric Polynomial Degree
## 2     scale numeric             Scale
## 3         C numeric              Cost
```

```r
# Radial basis kernel
caret::getModelInfo("svmRadial")$svmRadial$parameters
```

```
##   parameter   class label
## 1     sigma numeric Sigma
## 2         C numeric  Cost
```

---
class: center, middle

Not so .red[`purrr`]fect, now be gone!

```r
detach(package:purrr)
```

]

---

## Job attrition example

```r
# Load required packages
library(caret)    # for classification and regression training
library(dplyr)    # for data wrangling
library(ggplot2)  # for more awesome plotting
library(rsample)  # for attrition data and data splitting
library(pdp)      # for PDPs

*# Same setup as Naive Bayes module!

# Load the attrition data
attrition <- attrition %>%
  mutate(  # convert some numeric features to factors
    JobLevel = factor(JobLevel),
    StockOptionLevel = factor(StockOptionLevel),
    TrainingTimesLastYear = factor(TrainingTimesLastYear)
  )

# Train and test splits
set.seed(123)  # for reproducibility
split <- initial_split(attrition, prop = 0.7, strata = "Attrition")
trn <- training(split)
tst <- testing(split)
```

---
class: center, middle

## Choosing an evaluation metric is one of the most important tasks in any ML project!

---

## Job attrition example

```r
# Control params for SVM
ctrl <- trainControl(
  method = "cv", 
  number = 5, 
* classProbs = TRUE,
* summaryFunction = twoClassSummary
)

# Tune an SVM
set.seed(1854)  # for reproducibility
attr_svm <- train(
  Attrition ~ ., 
  data = trn,
* method = "svmRadial",
* preProcess = c("center", "scale"),
* metric = "ROC",
  trControl = ctrl,
  tuneGrid = data.frame(
    sigma = 0.008071434, 
    C = seq(from = 0.1, to = 5, length = 30)
  )
)
```

---

## Job attrition example

```r
# Plot tuning results
ggplot(attr_svm) + theme_light()
```

---

## Job attrition example

* .large[`caret`'s `varImp()` function provides a .red[*filter-based variable importance*] measure:]

* .large[`vip` has something similar, but it is still experimental!]

```r
# Filter-based variable 
# importance scores
plot(varImp(attr_svm))
```

]

]

---

## Job attrition example

```r
# Partial dependence plots
features <- c(
  "MonthlyIncome", 
  "TotalWorkingYears", 
  "OverTime", 
  "YearsAtCompany"
)
pdfs <- lapply(features, function(x) {
  autoplot(partial(attr_svm, pred.var = x, prob = TRUE)) +
    theme_light()
})
grid.arrange(
* grobs = pdfs,
  ncol = 2
)
```

]

]

---

## Job attrition example

```r
# Load required packages
*library(pROC)

# Plot train and test ROC curves
roc_trn <- roc(  # train AUC: 0.9717
  predictor = predict(attr_svm, newdata = trn, type = "prob")$Yes, 
  response = trn$Attrition,
  levels = rev(levels(trn$Attrition))
)
roc_tst <- roc(  # test AUC: 0.8567
  predictor = predict(attr_svm, newdata = tst, type = "prob")$Yes, 
  response = tst$Attrition,
  levels = rev(levels(tst$Attrition))
)
plot(roc_trn)
lines(roc_tst, col = "dodgerblue2")
legend("bottomright", legend = c("Train", "Test"), bty = "n", cex = 2.5,
       col = c("black", "dodgerblue2"), inset = 0.01, lwd = 2)
```

]

]

---
class: center, middle, inverse

background-image: url(Images/your-turn.jpg)
background-position: center
background-size: contain

???

Image credit: [imgflip](https://imgflip.com/)

---
class: middle

Retune the previous model using a polynomial kernel. What is the test ROC for your final model? Which features seem to be the most important?

**Hint:** Use .red[`method = "svmPoly"`] in the call to .red[`caret::train()`] and be sure to update the .red[`tuneGrid`] argument accordingly.

]

---
class: middle

```r
# Retune model using polynomial kenel
*set.seed(1720)  # for reproducibility
attr_svm_poly <- train(
  Attrition ~ ., 
  data = trn,
* method = "svmPoly",
* preProcess = c("center", "scale"),
* metric = "ROC",
  trControl = ctrl,
  tuneGrid = data.frame(
    degree = 1:3, 
    scale = 1,
    C = seq(from = 0.1, to = 5, length = 30)
  )
)
```

---
class: middle

```r
# Updated ROC curve
roc_tst_poly <- roc(  # test AUC: 0.8567
  predictor = predict(attr_svm_poly, newdata = tst, 
                      type = "prob")$Yes, 
  response = tst$Attrition,
  levels = rev(levels(tst$Attrition))
)
plot(roc_tst_poly, col = set1[1L])
lines(roc_tst, col = "dodgerblue2")
legend(
  x = "bottomright", 
  legend = c("Test (Poly)", "Test (RBF)"), 
  bty = "n", 
  cex = 1.5,
  col = set1[1L:2L], 
  inset = 0.01, 
  lwd = 2
)
```

---
class: middle, center

---

## Additional resources

<br/>

* .larger[[Fitting the entire regularization path](http://www.jmlr.org/papers/volume5/hastie04a/hastie04a.pdf)]

---

## Questions?