Table of Contents
A few years ago a Google Summer of Code project for Gentoo sough to Automatically generate[d] overlay of R packages and was undertaken by Andre Erdman resulting in the R_Overlay. Maintenance and development of the R_Overlay is now carried out bye Benda Xu.
Whilst some disagree that ebuilds R packages belong in portage (e.g. Denis Dupyren who it appears mentored the original project) this is a feature I have long sought as it makes installing and maintaining R packages across Gentoo systems as easy as copying /var/lib/portage/world
and is a bit simpler than maintaining an external script or code for setting up new (Gentoo) systems. I detail below the migration to using the R_Overlay.
Add the R_overlay
The R_Overlay is in layman, adding the overlay is trivial…
emerge -av layman layman -a R_overlay
Modify /etc/portage/repos.conf/layman.conf
to add the sync-type
and turn on automatic synchornisation…
[R_Overlay] priority = 50 location = /var/lib/layman/R_Overlay layman-type = rsync # ToDo - reconcile layman-type and git repository # sync-uri = https://anongit.gentoo.org/git/proj/R_overlay.git # sync-type = git auto-sync = Yes
The overlay will now be updated when you use eix-sync
(or emerge –sync
which is called in the background by eix-sync
).
dev-lang/R minimal
Initial forays led to conflicts with some packages requiring USE=minimal dev-lang/R
and others requiring USE=-minimal dev-lang/R
. After mailing the Gentoo Science mailing list Benda Xu, the current maintainer of R_Overlay, advised that ''USE=minimal'' should be the correct approach because USE=-minimal
builds R with a few recommended packages packages which are sometimes outdated and will not always meet the requirements of other packages from CRAN.
If you do not already have USE=minimal dev-lang/R
installed then add the appropriate USE flag and re-emerge…
echo 'dev-lang/R minimal' >> /etc/portage/package.use/dev-lang emerge -av dev-lang/R
Install Packages
R and R_Overlay are now installed and you can install R packages directly using portage. What these will be are highly user dependent so I can only demonstrate what I have installed and the problems I encountered, but the following installs the tidyverse meta-package which pulls in a large number of very useful packages along with the recommended packages, various GGplot2 extensions and a number of popular modelling packages.
# Tidyverse... emerge -av tidyverse # ggplot2 extensions emerge -av ggridges ggraph ggrepel ggnetwork ggmosaic gglasso ggimage ggforce gge ggalt ggROC ggHorizon ggExtra ggbeeswarm # Recommended (not necessarily the recommend version though, will pull in latest from CRAN) emerge -av KernSmooth MASS Matrix boot class sci-CRAN/cluster sci-CRAN/codetools foreign lattice mgcv nlme nnet rpart spatial survival # Modelling emerge -av lme4 glmnet Hmisc rms epitools # Sample Size emerge -av pwr TrialSize longpower simglm clusterPower samplesize PowerTOST # Miscellaneous emerge -av blockrand bookdown sci-CRAN/shiny rmarkdown
Manually Install Failed Packages
For various reasons a number of packages fail to install either directly or because of required dependencies (more on this below). These can be manually installed from R (as root) using…
# Install remaining tidyverse packages and dependancies install.packages(c("hms", "forcats", "xml2", "lubridate", "selectr", "mnormt", "rematch", "foreign", "readr", "rvest", "cellranger", "nlme", "readxl", "haven", "psych", "broom", "modelr", "tidyverse")) # Install remaining ggplot2 extensions and dependancies install.packages(c("ggraph", "gge", "ggimage", "ggraph")) # Install remaining recommended packages and dependancies install.packages(c("KernSmooth", "MASS", "Matrix", "boot", "class", "cluster", "codetools", "foreign", "lattice", "mgcv", "nlme", "nnet", "rpart", "spatial", "survival")) # Install remaining modelling packages and dependancies install.packages(c("lme4", "glmnet", "Hmisc", "rms", "epitools")) # Install remaining sample size packages and dependancies # Miscellaneous install.packages(c("blockrand", "bookdown", "shiny", "rmarkdown"))
SystemRequirement problems
I found that not every package on CRAN is available in R_Overlay (e.g. blogdown) and that in some instances only ebuilds for old package versions were included in ROverlay that did not meet the requirements of other packages. For example two dependencies of the tidyverse are as of writing xml2-1.2.0 and lubridate-1.7.3 but the versions in Roverlay are 0.1.2 and 1.7.0 respectively.
Both of these issues have a common root in that it is due to the packages specified SystemRequirements
, in the first instance blogdown requires Hugo framework which isn't currently in Portage, whilstxml2 has dependencies on libxml2 that could not be parsed when generating the ebuild. Benda has asked if would contact R developers to request moving towards machine-readable definitions of SystemRequirements
in the DESCRIPTION
file of R packages which are currently plain text and not automatically checked by packages themselves when installing. This is something that would need to be resolved upstream in R packages for R_Overlay to work well (hence Benda's request).
Not There Yet
As you can see quite a number of problems that are yet to be overcome still exist in using R_Overlay, but that an automated overlay exists is a HUGE step forward to R packages being handled in the same manner as Perl or Python modules.
Outdated Builds
These are likely down to the above issues with SystemRequirements
not being parsed, the following package sources could not be retrieved.
# grep '404 Not Found' /var/tmp/portage/sci-CRAN/*/temp/build.log -B3 | grep contrib /var/tmp/portage/sci-CRAN/ggHorizon-0.1/temp/build.log---2018-03-22 10:10:21-- http://cran.r-project.org/src/contrib/ggHorizon_0.1.tar.gz /var/tmp/portage/sci-CRAN/ggimage-0.0.2/temp/build.log---2018-03-22 10:10:35-- http://cran.r-project.org/src/contrib/ggimage_0.0.2.tar.gz /var/tmp/portage/sci-CRAN/igraph-1.1.2/temp/build.log---2018-03-22 10:09:30-- http://cran.r-project.org/src/contrib/igraph_1.1.2.tar.gz /var/tmp/portage/sci-CRAN/lubridate-1.7.0/temp/build.log---2018-03-26 11:15:41-- http://cran.r-project.org/src/contrib/lubridate_1.7.0.tar.gz /var/tmp/portage/sci-CRAN/Rcpp-0.12.15/temp/build.log---2018-03-22 09:11:46-- http://cran.r-project.org/src/contrib/Rcpp_0.12.15.tar.gz /var/tmp/portage/sci-CRAN/rgl-0.95.1441/temp/build.log---2018-03-26 11:23:58-- http://cran.r-project.org/src/contrib/rgl_0.95.1441.tar.gz /var/tmp/portage/sci-CRAN/xml2-0.1.2/temp/build.log---2018-03-26 11:15:01-- http://cran.r-project.org/src/contrib/xml2_0.1.2.tar.gz
Chekcsums Failed
Some packages failed because the ''Checksum'' of the retrieved source package didn't match what was expected…
# grep 'checksum' -i /var/tmp/portage/sci-CRAN/*/temp/build.log -B60 | grep contrib /var/tmp/portage/sci-CRAN/Matrix-1.2.12/temp/build.log---2018-03-26 12:34:13-- http://cran.r-project.org/src/contrib/Matrix_1.2-12.tar.gz /var/tmp/portage/sci-CRAN/mvtnorm-1.0.7/temp/build.log---2018-03-26 12:33:25-- http://cran.r-project.org/src/contrib/mvtnorm_1.0-7.tar.gz /var/tmp/portage/sci-CRAN/nloptr-1.0.4/temp/build.log---2018-03-26 12:33:05-- http://cran.r-project.org/src/contrib/nloptr_1.0.4.tar.gz
File Not Found
A number of packages failed after installation within R had completed when Portage could not find a file. The output looks like…
* Package: sci-CRAN/abind-1.4.5 * Repository: R_Overlay * USE: abi_x86_64 amd64 elibc_glibc kernel_linux userland_GNU * FEATURES: preserve-libs sandbox userpriv usersandbox * installing *source* package ‘abind’ ... ** package ‘abind’ successfully unpacked and MD5 sums checked ** R ** inst ** preparing package for lazy loading ** help *** installing help indices ** building package indices ** testing if installed package can be loaded * DONE (abind) Traceback (most recent call last): File "/usr/lib/portage/python3.5/doins.py", line 611, in <module> sys.exit(main(sys.argv[1:])) File "/usr/lib/portage/python3.5/doins.py", line 602, in main os.path.dirname(source)): File "/usr/lib/portage/python3.5/doins.py", line 450, in _doins return install_runner.install_file(source, os.path.dirname(dest)) File "/usr/lib/portage/python3.5/doins.py", line 386, in install_file return self._ins_runner.run(source, dest_dir) File "/usr/lib/portage/python3.5/doins.py", line 195, in run sstat = os.stat(source) FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/abind-1.4.5/work/abindy' * ERROR: sci-CRAN/abind-1.4.5::R_Overlay failed (install phase): * doins failed * * If you need support, post the output of `emerge --info '=sci-CRAN/abind-1.4.5::R_Overlay'`, * the complete build log and the output of `emerge -pqv '=sci-CRAN/abind-1.4.5::R_Overlay'`. * The complete build log is located at '/var/tmp/portage/sci-CRAN/abind-1.4.5/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/sci-CRAN/abind-1.4.5/temp/environment'. * Working directory: '/var/tmp/portage/sci-CRAN/abind-1.4.5/work/abind-1.4.5' * S: '/var/tmp/portage/sci-CRAN/abind-1.4.5/work/abind-1.4.5'
…and the packages that failed in this manner are…
# grep 'FileNotFound' /var/tmp/portage/sci-CRAN/*/temp/build.log /var/tmp/portage/sci-CRAN/abind-1.4.5/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/abind-1.4.5/work/abindy' /var/tmp/portage/sci-CRAN/acepack-1.4.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/acepack-1.4.1/work/acepacky' /var/tmp/portage/sci-CRAN/backports-1.1.2/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/backports-1.1.2/work/backportsy' /var/tmp/portage/sci-CRAN/bitops-1.0.6-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/bitops-1.0.6-r1/work/bitopsy' /var/tmp/portage/sci-CRAN/blockrand-1.3-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/blockrand-1.3-r1/work/blockrandy' /var/tmp/portage/sci-CRAN/boot-1.3.20/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/boot-1.3.20/work/booty' /var/tmp/portage/sci-CRAN/class-7.3.14/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/class-7.3.14/work/classy' /var/tmp/portage/sci-CRAN/cluster-2.0.6/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/cluster-2.0.6/work/clustery' /var/tmp/portage/sci-CRAN/codetools-0.2.15/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/codetools-0.2.15/work/codetoolsy' /var/tmp/portage/sci-CRAN/cubature-1.3.11/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/cubature-1.3.11/work/cubaturey' /var/tmp/portage/sci-CRAN/epitools-0.5.10/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/epitools-0.5.10/work/epitoolsy' /var/tmp/portage/sci-CRAN/evaluate-0.10.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/evaluate-0.10.1/work/evaluatey' /var/tmp/portage/sci-CRAN/fftwtools-0.9.8/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/fftwtools-0.9.8/work/fftwtoolsy' /var/tmp/portage/sci-CRAN/forcats-0.3.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/forcats-0.3.0/work/forcatsy' /var/tmp/portage/sci-CRAN/foreign-0.8.69/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/foreign-0.8.69/work/foreigny' /var/tmp/portage/sci-CRAN/Formula-1.2.2/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/Formula-1.2.2/work/Formulay' /var/tmp/portage/sci-CRAN/ggalt-0.4.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggalt-0.4.0/work/ggalty' /var/tmp/portage/sci-CRAN/ggbeeswarm-0.6.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggbeeswarm-0.6.0/work/ggbeeswarmy' /var/tmp/portage/sci-CRAN/ggExtra-0.7/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggExtra-0.7/work/ggExtray' /var/tmp/portage/sci-CRAN/ggforce-0.1.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggforce-0.1.1/work/ggforcey' /var/tmp/portage/sci-CRAN/gglasso-1.4/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/gglasso-1.4/work/gglassoy' /var/tmp/portage/sci-CRAN/ggmosaic-0.1.2/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggmosaic-0.1.2/work/ggmosaicy' /var/tmp/portage/sci-CRAN/ggnetwork-0.5.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggnetwork-0.5.1/work/ggnetworky' /var/tmp/portage/sci-CRAN/ggrepel-0.7.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggrepel-0.7.0/work/ggrepely' /var/tmp/portage/sci-CRAN/ggridges-0.4.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggridges-0.4.1/work/ggridgesy' /var/tmp/portage/sci-CRAN/ggROC-1.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/ggROC-1.0/work/ggROCy' /var/tmp/portage/sci-CRAN/highr-0.6/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/highr-0.6/work/highry' /var/tmp/portage/sci-CRAN/hms-0.4.2/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/hms-0.4.2/work/hmsy' /var/tmp/portage/sci-CRAN/iterators-1.0.9/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/iterators-1.0.9/work/iteratorsy' /var/tmp/portage/sci-CRAN/jpeg-0.1.8-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/jpeg-0.1.8-r1/work/jpegy' /var/tmp/portage/sci-CRAN/KernSmooth-2.23.15/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/KernSmooth-2.23.15/work/KernSmoothy' /var/tmp/portage/sci-CRAN/lattice-0.20.35/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/lattice-0.20.35/work/latticey' /var/tmp/portage/sci-CRAN/latticeExtra-0.6.28/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/latticeExtra-0.6.28/work/latticeExtray' /var/tmp/portage/sci-CRAN/locfit-1.5.9.1-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/locfit-1.5.9.1-r1/work/locfity' /var/tmp/portage/sci-CRAN/markdown-0.8/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/markdown-0.8/work/markdowny' /var/tmp/portage/sci-CRAN/MASS-7.3.49/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/MASS-7.3.49/work/MASSy' /var/tmp/portage/sci-CRAN/MatrixModels-0.4.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/MatrixModels-0.4.1/work/MatrixModelsy' /var/tmp/portage/sci-CRAN/minqa-1.2.4/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/minqa-1.2.4/work/minqay' /var/tmp/portage/sci-CRAN/mnormt-1.5.5/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/mnormt-1.5.5/work/mnormty' /var/tmp/portage/sci-CRAN/nipals-0.4/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/nipals-0.4/work/nipalsy' /var/tmp/portage/sci-CRAN/nlme-3.1.131.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/nlme-3.1.131.1/work/nlmey' /var/tmp/portage/sci-CRAN/nnet-7.3.12/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/nnet-7.3.12/work/nnety' /var/tmp/portage/sci-CRAN/png-0.1.7-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/png-0.1.7-r1/work/pngy' /var/tmp/portage/sci-CRAN/polspline-1.1.12/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/polspline-1.1.12/work/polspliney' /var/tmp/portage/sci-CRAN/proto-1.0.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/proto-1.0.0/work/protoy' /var/tmp/portage/sci-CRAN/pwr-1.2.2/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/pwr-1.2.2/work/pwry' /var/tmp/portage/sci-CRAN/rematch-1.0.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/rematch-1.0.1/work/rematchy' /var/tmp/portage/sci-CRAN/rpart-4.1.13/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/rpart-4.1.13/work/rparty' /var/tmp/portage/sci-CRAN/rstudioapi-0.7/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/rstudioapi-0.7/work/rstudioapiy' /var/tmp/portage/sci-CRAN/samplesize-0.2.4/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/samplesize-0.2.4/work/samplesizey' /var/tmp/portage/sci-CRAN/selectr-0.3.2/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/selectr-0.3.2/work/selectry' /var/tmp/portage/sci-CRAN/shiny-1.0.5/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/shiny-1.0.5/work/shinyy' /var/tmp/portage/sci-CRAN/SparseM-1.77/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/SparseM-1.77/work/SparseMy' /var/tmp/portage/sci-CRAN/spatial-7.3.11/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/spatial-7.3.11/work/spatialy' /var/tmp/portage/sci-CRAN/TeachingDemos-2.10/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/TeachingDemos-2.10/work/TeachingDemosy' /var/tmp/portage/sci-CRAN/tiff-0.1.5-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/tiff-0.1.5-r1/work/tiffy' /var/tmp/portage/sci-CRAN/TrialSize-1.3-r1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/TrialSize-1.3-r1/work/TrialSizey' /var/tmp/portage/sci-CRAN/viridis-0.5.0/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/viridis-0.5.0/work/viridisy' /var/tmp/portage/sci-CRAN/zoo-1.8.1/temp/build.log:FileNotFoundError: [Errno 2] No such file or directory: b'/var/tmp/portage/sci-CRAN/zoo-1.8.1/work/zooy'
What is common across all of these is that all the missing files have a y
appended to them which may be down to a bug in the way the ebuilds for CRAN packages are automatically generated.