Compilation on MacOS Monterey M1 (ARM64 architecture)

[aourednik]

Compiling from source under MacOS Monterey 12.0.1 on M1 (ARM64) fails.
R version 4.1.2 (2021-11-01)
Platform: aarch64-apple-darwin20.6.0 (64-bit)

Steps :

• reinstall Homebrew for ARM64 (instead of usr/local/ this installs homebrew to the arm64 compatible directory opt/homebrew)
• install pkg-config, GDAL, PROJ and GEOS with homebrew install
• add these lines to ~/R./Makevars:

FC      = /opt/homebrew/bin/gfortran
F77     = /opt/homebrew/bin/gfortran

• run install.packages("sf") : Fails with error "gdal-config not found or not executable"
• run install.packages("sf",configure.args = "--with-gdal-config=/opt/homebrew/opt/gdal/bin/gdal-config") : Solves the gdal-problem, but also fails with this trace:

...
checking GDAL: linking with --libs only... yes
checking GDAL: /opt/homebrew/Cellar/gdal/3.3.3/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... yes
configure: GDAL: 3.3.3
./configure: line 3657: pkg-config: command not found
checking proj.h usability... yes
checking proj.h presence... yes
checking for proj.h... yes
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘sf’

[aourednik]

This is possibly solved here by @drkrynstrng :
rspatial/terra#396
But has this been merged?

[abuabara]

on the M1 I have been using it without homebrew. You need to install some of these libraries: https://mac.r-project.org/libs-arm64/

i'm late for a class ... i will try to reply you later with more information. Just wanted to give you some update.

[abuabara]

On my M1 I just had to install "gfortran" and "pkgconfig" here "/opt/R/arm64/bin". No homebrew was necessary. The packages in R compiles from the source normally after that. MacOS may ask you permissions, just open the files, right click on it, and terminal will allow permissions automatically.

By the way, "to install" these libraries, just unzip the files and move the folders, keeping the same structure. The very first one you will only need to move the folders. If you need more libraries, keep the same file structure.

Hope it helps. Let us know if any question.

[edzer]

close?

[aourednik]

Now it compiles correctly on an M1 with Monterey even with a standard homebrew install of pkg-config, GDAL, PROJ and GEOS.
Perhaps also due to today's series of updates to many other R libraries?
In any case, it compiles.
Many thanks!

[CCheCastaldo]

Hi @aourednik. It seems as though you got sf to install from source with pkg-config, gdal, proj, and geos installed via homebrew on an M1 Mac. I have the same setup as you and am still getting the original error above (pkg-config not found). Were there additional steps you had to take? I did:

install.packages("sf",configure.args = "--with-gdal-config=/opt/homebrew/bin/gdal-config", type = "source")

and get the same error above:

...
checking GDAL: linking with --libs only... yes
checking GDAL: /opt/homebrew/Cellar/gdal/3.3.3_1/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... yes
configure: GDAL: 3.3.3
./configure: line 3662: pkg-config: command not found
checking proj.h usability... no
checking proj.h presence... no
checking for proj.h... no
checking proj_api.h usability... no
checking proj_api.h presence... no
checking for proj_api.h... no

and the same error again after running this in R prior to the install.packages command:

print(Sys.setenv(PKG_CONFIG_PATH = "/opt/homebrew/lib/pkgconfig/"))
print(Sys.setenv(PKG_CONFIG = "/opt/homebrew/bin/pkg-config"))

I have no problem executing the pkg-config command in a bash terminal. For example, which finds it just fine and pkg-config --modversion gdal finds the gdal .pc file to return its version. It just seems as though R doesn't know where pkg-config is and I cannot seem to tell it. I left off the .Makevars bit you did as I am not sure if this is needed.

Thanks for any advice.

MacOS Monterey 12.1 on M1 Pro (ARM64)
R version 4.1.2 (2021-11-01)

[CCheCastaldo]

OK, I think I figured it out. For whatever reason, R was not recognizing the PATH env variable updated in my .bash_profile to begin with /opt/homebrew/bin. Instead R was defaulting to the non-updated PATH that begins with usr/local/bin. This was causing R to not find pkg-config during installations of sf from source. This then worked for me:

1. brew install pkg-config, gdal, and proj
2. In R, adjust the path to point first to /opt/homebrew/bin

Sys.getenv("PATH")
path_temp <- paste0("/opt/homebrew/bin:", Sys.getenv("PATH"))
print(Sys.setenv(PATH = path_temp))
Sys.getenv("PATH")

3. Install rgdal and sf from source.

install.packages("rgdal", type = "source")
install.packages("sf", type = "source")

I did not need to add any configuration arguments for gdal or proj, as R found everything and both packages installed cleanly (although sf had 6 warnings which I have seen in the past on previous installs). Now sf links to the homebrew spatial libraries:

library(sf)
Linking to GEOS 3.10.1, GDAL 3.3.3, PROJ 8.2.0; sf_use_s2() is TRUE

This same approach also worked for terra and stars. I am not sure if the PATH always needs to have /opt/homebrew/bin in front or just when installing these packages, but that should be obvious after some use. Hope this is helpful and does not add confusion if I am wrong about all this.

My system is:
MacOS Monterey 12.1 on M1 Pro (ARM64)
R version 4.1.2 (2021-11-01) -- "Bird Hippie"
Platform: aarch64-apple-darwin20 (64-bit)

[aourednik]

Unfortunately, the problem is back with gdal 3.4 : sf is looking for the 2.9 version.

As of 2022, Homebrew updates GDAL to 3.4 /opt/homebrew/Cellar/gdal/3.4.0/lib/libgdal.30.dylib , symlinked from /opt/homebrew/opt/gdal/

The effect on the sf library is immediate:

> library("sf")
Erreur : le chargement du package ou de l'espace de noms a échoué pour ‘sf’ in dyn.load(file, DLLpath = DLLpath, ...) :
impossible de charger l'objet partagé '/opt/homebrew/lib/R/4.1/site-library/sf/libs/sf.so':
  dlopen(/opt/homebrew/lib/R/4.1/site-library/sf/libs/sf.so, 0x0006): Library not loaded: /opt/homebrew/opt/gdal/lib/libgdal.29.dylib
  Referenced from: /opt/homebrew/lib/R/4.1/site-library/sf/libs/sf.so
  Reason: tried: '/opt/homebrew/opt/gdal/lib/libgdal.29.dylib' (no such file), '/opt/homebrew/Cellar/r/4.1.2/lib/R/lib/libgdal.29.dylib' (no such file), '/opt/homebrew/Cellar/gdal/3.4.0/lib/libgdal.29.dylib' (no such file), '/opt/homebrew/Cellar/r/4.1.2/lib/R/lib/libgdal.29.dylib' (no such file)

Trying to reinstall:

> remotes::install_github("r-spatial/sf", configure.args = "--with-gdal-config=/opt/homebrew/opt/gdal/bin/gdal-config --with-sqlite3-lib=/opt/homebrew/opt/sqlite3")
Downloading GitHub repo r-spatial/sf@HEAD
   checking for file ‘/private/var/folders/p8/7m7_z7754pdffd5z_f7n15540000gp/T/R✔  checking for file ‘/private/var/folders/p8/7m7_z7754pdffd5z_f7n15540000gp/T/RtmpADyrFG/remotes1208caa9cfe8/r-spatial-sf-e2c8936/DESCRIPTION’
─  preparing ‘sf’:
✔  checking DESCRIPTION meta-information ...
─  cleaning src
─  running ‘cleanup’
─  checking for LF line-endings in source and make files and shell scripts (443ms)
─  checking for empty or unneeded directories
─  building ‘sf_1.0-6.tar.gz’
   
Installation du package dans ‘/opt/homebrew/lib/R/4.1/site-library’
(car ‘lib’ n'est pas spécifié)
* installing *source* package ‘sf’ ...
** using staged installation
configure: CC: clang
configure: CXX: clang++ -std=gnu++11
configure: gdal-config set to /opt/homebrew/opt/gdal/bin/gdal-config
checking gdal-config exists... yes
checking gdal-config executable... yes
checking gdal-config usability... yes
configure: GDAL: 3.4.0
checking GDAL version >= 2.0.1... yes
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ISO C89... none needed
checking how to run the C preprocessor... clang -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking gdal.h usability... yes
checking gdal.h presence... yes
checking for gdal.h... yes
checking GDAL: linking with --libs only... yes
checking GDAL: /opt/homebrew/Cellar/gdal/3.4.0/share/gdal/pcs.csv readable... no
checking GDAL: checking whether PROJ is available for linking:... yes
checking GDAL: checking whether PROJ is available fur running:... yes
configure: GDAL: 3.4.0
configure: pkg-config proj exists, will use it
configure: using proj.h.
configure: PROJ: 8.2.1
checking PROJ: checking whether PROJ and sqlite3 are available for linking:... no
configure: error: libproj or sqlite3 not found in standard or given locations.
ERROR: configuration failed for package ‘sf’

I've tried to install an older version of GDAL with homebrew, to no avail:

brew install gdal@2
Warning: No available formula with the name "gdal@2". Did you mean gdal or gdl?
==> Searching for similarly named formulae...
These similarly named formulae were found:
gdal ✔                                   gdl

[rsbivand]

Please note that homebrew is the choice and complete responsibility of anyone choosing that path. M1 is supported with CRAN binaries, static linked to external software (usually not the latest version, but fairly new). You seem to be surprised that your sf installation ceased to load when the dynamically linked library against which it was built and installed is absent. It is unusual for ABI compatibility to extend over GDAL/PROJ/GEOS releases, so any local update to any of these upstream dependencies will require a source install of sf.

The final problem you hit is that software required by PROJ is not found (PROJ needs libtiff, libsqlite3 and libcurl in places where they can be found by configure). So for convenience and less loss of time, use CRAN binaries only and never install spatial packages using external software from source (sf, lwgeom, terra, vapour, rgdal, rgeos, etc.). The only viable reason for source installs using non-CRAN dynamic linking is that newer versions offer functionality not present in CRAN static-linked binaries.

You may of course install source packages using CRAN static-built libraries, but this is only relevant if you must have the development version of a package, but this should really only apply to those contributing to package development (you've made an edit to a local fork of sf and want to check its performance).

[aourednik]

Thanks for fast reply @rsbivand .

Today I've noticed that my MacOS version of R, for a reason I believed necessary up to now, defaults to building from source when running install.packages("package-name") despite there being MacOS binaries available on CRAN.
I confirm that just downloading the .tgz for arm64 from CRAN and running R CMD INSTALL sf_1.0-5.tgz from the MacOS terminal solves the problem.

SF is a great package, I've become quite dependent to it in my processes, it's always a haunting experience when it stops working :-)

[abuabara]

just have a .Renviron file in your home folder with R_COMPILE_AND_INSTALL_PACKAGES=never and don't worry anymore about compiling packages from source. Only 2 cents from a non-developer user.

[aourednik]

Thanks @abuabara, this could be a general workaround indeed. But for some cases, I need packages to compile, for instance data.table, that needs a special Makevars file with CC and CXX flags specifying the use of fopenmp.

However, I've found a one-liner to solve all the sf installation issues above, forever :-) Run:

install.packages("https://cran.r-project.org/bin/macosx/big-sur-arm64/contrib/4.1/sf_1.0-5.tgz", repos = NULL, type = .Platform$pkgType)

instead of install.packages("sf")

@rsbivand : perhaps this could be noted somewhere in the installation section of the sf website ?

I've also filed a feature proposal to RStudio to make this a default behaviour of package manager: rstudio/rstudio#10360

[rsbivand]

Thanks! See #1621 (comment)

[rundel]

I just ran into this issue as well installing sf on a new M1 machine and I think I have any idea about the cause and a potential fix (for at least the original issue).

With homebrew on these newer Macs everything is being installed into /opt/homebrew/ with libraries in /opt/homebrew/Cellar/<lib>/<ver>/lib and then symlinks in /opt/homebrew/lib. The issue here is that the core libraries are getting picked up with the necessary -L flags from the config tools (e.g. geos-config) but these are for the library specific directory and not the general /opt/homebrew/lib directory. So if you look at $LIBS specifically it has:

LIBS='-L/opt/homebrew/Cellar/proj/8.2.1/lib -lproj -lstdc++ -lsqlite3 -lpthread -ltiff -lcurl -lm -ldl    -L/opt/homebrew/Cellar/gdal/3.4.1_1/lib -lgdal  -L/opt/homebrew/Cellar/gdal/3.4.1_1/lib -lgdal'

which doesn't include -L/opt/homebrew/lib for the linker and hence it cant find the ancillary libraries (in my case it was libtiff that was causing the specific failure).

However, this information is available in $LDFLAG and so one possible fix is to add this to

sf/configure.ac

Line 441 in cc698f2

${CXX} ${CPPFLAGS} -o proj_conf_test proj_conf_test.cpp ${LIBS} $SQLITE3_LIBS -lsqlite3 2> errors.txt

and

sf/configure.ac

Line 467 in cc698f2

${CC} ${CFLAGS} ${CPPFLAGS} -o proj_conf_test proj_conf_test.c ${PROJ_LIBS}

which then includes the necessary -L/opt/homebrew/lib and all of the libraries to be found allowing linking to succeed for these checks.

One non-package interim fix is that users can also add LIBRARY_PATH=/opt/homebrew/lib to their .Renviron which then allows the package to build correctly as the linker will then search that path by default.

@rsbivand @edzer

[rsbivand]

I think that only non-package resolutions are acceptable. The draft Spatial task view (private repo until new task views are rolled out) says:

Installation of packages like r pkg("sf") and r pkg("terra") which use
external software libraries such as PROJ, GDAL or GEOS requires care. For
most users on platforms such as Windows or macOS who are not themselves
package developers, it is always better to avoid what are known as
source installs, because CRAN binary packages include all of the external
software required. Because getOption("pkgType") on these platforms is
usually "both", you may be asked to choose to install a source package
if it is more recent than the latest binary.

Please do not be tempted to choose a source install for r pkg("sf") or
r pkg("terra") or similar; the binary package will be generated within a
day or two. To avoid being asked, you may see from ?options
under options provided by the utils package that the default
behaviour of your installation of R may be controlled
by setting options install.packages.check.source and
install.packages.compile.from.source, or by setting environment variable
R_COMPILE_AND_INSTALL_PACKAGES, see also this helpful comment.

If you are a developer using Windows or macOS or installing
from github, the same static-linked
binary external software libraries, header files, etc. as those used in
building CRAN binary packages are available from: Windows 4.0 and 4.1
downloaded on-the-fly,
Windows 4.2 forthcoming rtools42
and macOS both architectures. These
external software libraries have been built using the same compile and
link settings as R itself, so avoid the risk of possible errors caused by
mismatched binaries.

If you are a user (or developer) on systems where getOption("pkgType")
is "source", you will need to ensure that the external software is
available when installing source packages. Advice for some such systems may
be found here.
The most common reason
for failure is having multiple versions of external software installed
on your platform.

It will be much more helpful if homebrew people (who have by definition some insight into build systems) rather help with migrating GEOS, PROJ and especially GDAL to static cmake for the static binaries. Incidentally, shall I open a PR to sf/README.md to remove homebrew completely, I don't think homebrew should be advocated so obviously when most homebrew issues actually suggest that the questioner should only be installing binary packages.

Some work has been done by Simon U, Tomas K and especially BDR, but GDAL is very likely to be very difficult to resolve. Contributions to get Rtools42 and the macOS recipes ready for GDAL 3.5 (see also this thread: https://lists.osgeo.org/pipermail/gdal-dev/2022-January/055253.html) would be most valuable, as cmake implementations in these libraries are most likely happening because their downstream users (QGIS, PostGIS) want it, not because library developers know cmake.

[stineb]

• ~/R./Makevars:

Under 'Steps', I believe this should be

~/.R/Makevars:

And not

~/R./Makevars:

(Note the dot before, not after the 'R'.)

[JosiahParry]

@abuabara, using the binaries at https://mac.r-project.org/bin/darwin20/arm64/ how did you install them?

[rsbivand]

I think https://mac.r-project.org/bin/ is where to start, you'll need the support files and headers too. I can check Thursday afternoon CET.

[JosiahParry]

I'll update my version of R and try from scratch and do my best to document it.

[JosiahParry]

Installed a fresh R version 4.2.2. CRAN binaries have been installed but are not picked up by {sf} when installing. There is no other gdal installation.

which gdal
gdal not found

In terminal

sudo R
source("https://mac.R-project.org/bin/install.R")
install.libs("gdal")
Downloading https://mac.R-project.org/bin/REPOS ...
Using repository  https://mac.R-project.org/bin/darwin20/arm64 ...
Downloading index  https://mac.R-project.org/bin/darwin20/arm64/PACKAGES ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/gdal-3.5.3-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/pkgconfig-0.29.2-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/xz-5.2.6-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/libpq-14.5-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/proj-9.1.0-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/openjpeg-2.5.0-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/libwebp-1.2.4-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/libgeotiff-1.7.1-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/freexl-1.0.6-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/sqlite3-3.39.3-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/expat-2.4.8-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/hdf4-4.2.15-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/hdf5-1.12.2-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/szip-2.1.1-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/netcdf-4.8.1-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/geos-3.11.0-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/unixodbc-2.3.11-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/zlib-stub-0.1-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/tiff-4.4.0-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/libpng-1.6.38-darwin.20-arm64.tar.xz ...
Downloading + installing  https://mac.R-project.org/bin/darwin20/arm64/jpeg-9e-darwin.20-arm64.tar.xz ...
quit()


ls /opt/R/arm64/bin | grep "gdal"
gdal-config
gdal_contour
gdal_create
gdal_grid
gdal_rasterize
gdal_translate
gdal_viewshed
gdaladdo
gdalbuildvrt
gdaldem
gdalenhance
gdalinfo
gdallocationinfo
gdalmanage
gdalmdiminfo
gdalmdimtranslate
gdalsrsinfo
gdaltindex
gdaltransform
gdalwarp

Open fresh RStudio session:

devtools::load_all()

── R CMD INSTALL ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
─  installing *source* package ‘sf’ ...
   ** using staged installation
   configure: CC: clang -arch arm64
   configure: CXX: clang++ -arch arm64 -std=gnu++14
   checking for gdal-config... no
   no
   configure: error: gdal-config not found or not executable.
   ERROR: configuration failed for package ‘sf’
─  removing ‘/private/var/folders/wd/xq999jjj3bx2w8cpg7lkfxlm0000gn/T/RtmpvnkkEc/devtools_install_2a8c9ab8d8/sf’
Error in `(function (command = NULL, args = character(), error_on_status = TRUE, …`:
! System command 'R' failed
---

sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.0

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

[edzer]

is /opt/R/arm64/bin in your search path? If you don't have that and don't want that, try in the sf directory to run configure -h to see options to configure to point it to the place where gdal-config etc. are found.

[JosiahParry]

I added /opt/R/arm64/bin to my path. Got much much further along in the installation (not without finding a bug in RStudio itself). The configure command is not available on my machine. I'm guessing that's something else that needs to be installed.

I was able to get sf to build locally like so

1. Fresh install of R
2. from the terminal sudo R
   • sudo is necessary because otherwise the requisite folders cannot be created and written to
3. source("https://mac.R-project.org/bin/install.R")
4. install.libs(“gdal”)
   • Note: I also installed proj but I'm unsure if it was necessary
5. install.libs("zstd")
   • this library was missing and was required. after installation compilation completed
6. ensure that /opt/R/arm64/bin is on the path.

This uses only CRAN binaries.

[edzer]

The configure command is not available on my machine.

configure is a script in the sf source directory.

[JosiahParry]

Ah ./configure -h :)