In the past, I've used the R.exe commandline utility to install R packages for use with SQL Server's Machine Learning Services. As with many things, there's more than one way to accomplish this task. I've come to realize it can be done using RStudio. This is now my preferred method. As I recall, my RStudio installations included a local installation of the R runtime, which is not what Machine Learning Services uses. For SQL Server 2016-2019, the R runtime is installed as part of the SQL Server installation.
So, with RStudio installed on the SQL Server host, you'll need to change the R Version/Installation for RStudio. Go to Tools on the menu and select Global Options. Select the R General page/option on the left and click the Change... button for R Sessions.
A second "Choose R Installation" dialog will appear--click the Browse... button and navigate to the folder where R is installed for your SQL Server instance. (My folder was "C:\Program Files\Microsoft SQL Server\MSSQL15.SQL2019\R_SERVICES".)
To install packages, you may need RTools, which is for building R packages from source on Windows. RTools is a separate installation, and the version you install depends on the version of the R runtime. My R version (for SQL Server 2019 Machine Learning Services) is 3.5.2, so I downloaded and installed RTools version 3.5.
From here, it's all R code. A package I installed recently (with some difficulty) was the arrow package. The syntax I used for installing packages is as follows:
install.packages("arrow", dependencies=TRUE, repos='http://cran.rstudio.com/')
Running that one line of code took a very long time. By specifying dependencies=TRUE, many other dependent packages were installed in addition to arrow. Afterwards, there were four packages that failed to install, including arrow. I ran the line of code a second time. Three of the four packages succeeded this time. (Perhaps the dependencies needed to be installed in a specific order? Or maybe there were some circular references?) More than once, I was prompted with this question:
I clicked 'Yes' each time. But that one dependency wouldn't install. RTools apparently couldn't compile the source code. So I eventually clicked 'No' and the package installation succeeded (presumably with an older, pre-compiled binary.)
Another option for installing packages is a GUI within RStudio. Click Tools on the menu and Install Packages...
It looks fairly straightforward, although I haven't tried it myself. Still, I thought it worth mentioning as yet another option.