Compile issues as dynamic module with MUSL

[cchamplin]

I'm trying to get pagespeed to build as a dynamic module for nginx. I'm on Alpine which is a MUSL system.

I've built PSOL from source (with some adjustment to get it to compile).

When attempting to build the nginx module I'm getting a compilation error:

/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /home/test/nginx/clean/ngx_pagespeed-1.10.33.7-beta-dynamic//psol/lib/Release/linux/x64/pagespeed_automatic.a(142.deflate.o.o): relocation R_X86_64_PC32 against symbol 'pagespeed_ol_deflateResetKeep' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
objs/Makefile:1170: recipe for target 'objs/ngx_pagespeed.so failed

The PSOL was compiled with:
make all BUILDTYPE=Release CXXFLAGS=" -I/usr/include/apr-1 -I/home/test/nginx/libpng-1.2.56 -D_GLIBCXX_USE_CXX11_ABI=0 -Duse_system_libs=1 -Duse_system_icu=1" CFLAGS=" -I/usr/include/apr-1 -I/home/test/nginx/libpng-1.2.56 -D_GLIBCXX_USE_CXX11_ABI=0 -Duse_system_libs=1 -Duse_system_icu=1"

GCC -v

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/5.3.0/lto-wrapper
Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-5.3.0/configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --build=x86_64-alpine-linux-musl --host=x86_64-alpine-linux-musl --target=x86_64-alpine-linux-musl --with-pkgversion='Alpine 5.3.0' --enable-checking=release --disable-fixed-point --disable-libstdcxx-pch --disable-multilib --disable-nls --disable-werror --disable-symvers --enable-__cxa_atexit --enable-esp --enable-cloog-backend --enable-languages=c,c++,objc,java,fortran,ada --disable-libssp --disable-libmudflap --disable-libsanitizer --enable-shared --enable-threads --enable-tls --with-system-zlib
Thread model: posix
gcc version 5.3.0 (Alpine 5.3.0)

Any help getting this to build would be appreciated.

[morlovich]

Sounds like something like -fPIC may be needed for CXXFLAGS.

On Tue, Mar 29, 2016 at 11:53 AM, Caleb Champlin [email protected]
wrote:

I'm trying to get pagespeed to build as a dynamic module for nginx. I'm on
Alpine which is a MUSL system.

I've built PSOL from source (with some adjustment to get it to compile).

When attempting to build the nginx module I'm getting a compilation error:

/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld:
/home/test/nginx/clean/ngx_pagespeed-1.10.33.7-beta-dynamic//psol/lib/Release/linux/x64/pagespeed_automatic.a(142.deflate.o.o):
relocation R_X86_64_PC32 against symbol 'pagespeed_ol_deflateResetKeep' can
not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/5.3.0/../../../../x86_64-alpine-linux-musl/bin/ld:
final link failed: Bad value
collect2: error: ld returned 1 exit status
objs/Makefile:1170: recipe for target 'objs/ngx_pagespeed.so' failed

The PSOL was compiled with:
make all BUILDTYPE=Release CXXFLAGS=" -I/usr/include/apr-1
-I/home/test/nginx/libpng-1.2.56 -D_GLIBCXX_USE_CXX11_ABI=0
-Duse_system_libs=1 -Duse_system_icu=1" CFLAGS=" -I/usr/include/apr-1
-I/home/test/nginx/libpng-1.2.56 -D_GLIBCXX_USE_CXX11_ABI=0
-Duse_system_libs=1 -Duse_system_icu=1"

GCC -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-alpine-linux-musl/5.3.0/lto-wrapper

Target: x86_64-alpine-linux-musl
Configured with: /home/buildozer/aports/main/gcc/src/gcc-5.3.0/configure
--prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--build=x86_64-alpine-linux-musl --host=x86_64-alpine-linux-musl
--target=x86_64-alpine-linux-musl --with-pkgversion='Alpine 5.3.0'
--enable-checking=release --disable-fixed-point --disable-libstdcxx-pch
--disable-multilib --disable-nls --disable-werror --disable-symvers
--enable-__cxa_atexit --enable-esp --enable-cloog-backend
--enable-languages=c,c++,objc,java,fortran,ada --disable-libssp
--disable-libmudflap --disable-libsanitizer --enable-shared
--enable-threads --enable-tls --with-system-zlib
Thread model: posix
gcc version 5.3.0 (Alpine 5.3.0)

Any help getting this to build would be appreciated.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1166

[cchamplin]

Sorry I should have mentioned I've already tried building both PSOL and NGINX with -fPIC with no change

[morlovich]

Weird. Did you include it in CFLAGS as well?

On Tue, Mar 29, 2016 at 12:50 PM, Caleb Champlin [email protected]
wrote:

Sorry I should have mentioned I've already tried building both PSOL and
NGINX with -fPIC with no change

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1166 (comment)

[cchamplin]

Yup. The only thing I didn't do is compile PSOL with LDFLAGS -shared.

[morlovich]

Sorry, looks like I missed CFLAGS on your line above... I am stumped,
hopefully one my colleagues can chime in.

One other thing that's worth pointing out: Stuff
like: -Duse_system_libs=1 -Duse_system_icu=1" needs to be passed to gyp,
not to make --- passing it to make doesn't do anything.
(Though those haven't really been tested for psol.a, just mod_pagespeed,
anyway --- you might need to remove some .a files from the Makefile that
makes the .a, and add some -l
lines at nginx level).

That's actually sort of relevant since the object file you get the link
erorr on comes from zlib, which you probably intended to dynamic link
anyway.

On Tue, Mar 29, 2016 at 4:22 PM, Caleb Champlin [email protected]
wrote:

Yup. The only thing I didn't do is compile PSOL with LDFLAGS -shared.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1166 (comment)

[cchamplin]

Okay we can probably close this, it appears that your comment about removing the static libraries from the psol Makefile and adding them to the nginx compilation. Thanks for your help!

For anyone who comes across this in the future looking for information on how to get pagespeed to work on Alpine/MUSL here's a quick guide:

Grab and build a local version of libpng1.2

Download the PSOL source

Patch chromium to compile on MUSL (this is only needed until upstream chromium is used in PSOL, as these changes have already been applied in latest Chromium source)
See: https://codereview.chromium.org/1422623002/#ps40001 https://codereview.chromium.org/1514483008

psol-chromium.stacktrace.patch.txt

After it's been patched compile mod_pagespeed:
make BUILDTYPE=Release CXXFLAGS=" -I/usr/include/apr-1 -I/home/test/nginx/libpng-1.2.56 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" CFLAGS=" -I/usr/include/apr-1 -I/home/test/nginx/libpng-1.2.56 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0"

Move to pagespeed/automatic
Remove the following static libraries from the make file: libapr, libaprutil, libjpeg, libjpeg_turbo, libicudata, libicuuc, libpng, libopenssl, libzlib

Compile PSOL
make all BUILDTYPE=Release CXXFLAGS=" -I/usr/include/apr-1 -I/home/test/nginx/libpng-1.2.56 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" CFLAGS=" -I/usr/include/apr-1 -I/home/test/nginx/libpng-1.2.56 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0"

Once that's done you should have a pagespeed_automatic.a available for use, and you can now build nginx.

I built ngx_pagespeed as a dynamic module using the following:
./configure --with-ipv6 --with-file-aio --with-pcre-jit --with-http_dav_module --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --with-http_v2_module --with-http_auth_request_module --with-mail --with-mail_ssl_module --add-dynamic-module=/home/test/nginx/clean/ngx_pagespeed-1.10.33.7-beta-dynamic/ --with-cc-opt="-fPIC" --with-ld-opt="-luuid -lapr-1 -laprutil-1 -licudata -licuuc -lpng12 -lturbojpeg -ljpeg -fPIC"

[morlovich]

Did you also pass in the -D flags to gyp? It's potentially risky if you
don't, since it may end up using wrong header files (the ones for bundled
versions, not system ones) for things like the SSL library otherwise, which
may be problematic
if the two aren't binary compatible (likely an issue with system OpenSSL
vs. bundled BoringSSL).

On Wed, Mar 30, 2016 at 1:10 PM, Jeffrey Crowell [email protected]
wrote:

Closed #1166 #1166.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1166 (comment)

[ghost]

I am trying to build from an alpine container, I tried following the guide

I am stuck at:

CC(target) out/Release/obj.target/apr/third_party/apr/src/locks/unix/global_mutex.o
  CC(target) out/Release/obj.target/apr/third_party/apr/src/locks/unix/proc_mutex.o
third_party/apr/src/locks/unix/proc_mutex.c: In function 'proc_mutex_proc_pthread_create':
third_party/apr/src/locks/unix/proc_mutex.c:384:15: warning: implicit declaration of function 'pthread_mutexattr_setrobust_np' [-Wimplicit-function-declaration]
     if ((rv = pthread_mutexattr_setrobust_np(&mattr, 
               ^
third_party/apr/src/locks/unix/proc_mutex.c:385:48: error: 'PTHREAD_MUTEX_ROBUST_NP' undeclared (first use in this function)
                                                PTHREAD_MUTEX_ROBUST_NP))) {
                                                ^
third_party/apr/src/locks/unix/proc_mutex.c:385:48: note: each undeclared identifier is reported only once for each function it appears in
third_party/apr/src/locks/unix/proc_mutex.c: In function 'proc_mutex_proc_pthread_acquire':
third_party/apr/src/locks/unix/proc_mutex.c:440:13: warning: implicit declaration of function 'pthread_mutex_consistent_np' [-Wimplicit-function-declaration]
             pthread_mutex_consistent_np(mutex->pthread_interproc);
             ^
third_party/apr/apr.target.mk:394: recipe for target 'out/Release/obj.target/apr/third_party/apr/src/locks/unix/proc_mutex.o' failed
make: *** [out/Release/obj.target/apr/third_party/apr/src/locks/unix/proc_mutex.o] Error 1

tried compiling with

make AR.host="$PWD/build/wrappers/ar.sh" AR.target="$PWD/build/wrappers/ar.sh" BUILDTYPE=Release mod_pagespeed_test pagespeed_automatic_test CXXFLAGS=" -I/usr/include/apr-1 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" CFLAGS=" -I/usr/include/apr-1 -fPIC -D_GLIBCXX_USE_CXX11_ABI=0"

[iler]

@cchamplin What version of Nginx and ngx_pagespeed you have been using? I don't quite understand what patches should be applied? Those you have linked don't seem to match with the contents of mod_pagespeed repo.

[cchamplin]

@natostanco I'm not 100% on that one, APR was not directly built as part of my process, and I don't recall doing anything to prevent it from being built.

@iler I built against the prepackaged source available here apache/incubator-pagespeed-mod#968 specifically version 1.10.33.7

[cchamplin]

As an aside FYI I've been asked to try and put together and APK build of this whole thing, which hopefully I'll be able to get to some evening this week.

[iler]

@cchamplin Thanks. Will try with that. And +1 for APK build.

[iler]

@cchamplin Any chance you faced this problem https://gist.github.com./iler/f1e519168bace48545d494edb7b2baa6? I'm building Docker container and it's failing. You can see the build commands from my Dockerfile https://gist.github.com./iler/90a87a46b50ffe218417871288247afc

[ghost]

for that thing I just renamed __timezone to timezone in the file specified file. But icu should be provide by the system, to use system libs when syncing from gclient I had to export GYP_DEFINES="use_system_libs=1 ...etc" then sync with gclient, even tough it still downloads the src files it should use system libs

[cchamplin]

@iler

• You'll want to make sure you're passing use_system_libs=1 (specifically use_system_icu=1) to gyp.
• In pagespeed/automatic I believe you can probably get away with just make psol instead of make all.
• There are a few other files that need patching for this whole thing to work which I forgot in my original post (sorry!). The patch I attached to this file is also slightly wrong (but unlikely to actually cause any issues in this situation)
• You'll also need to bring over PSOL content (necessary includes and pagespeed_automatic.a) into your ngx_pagespeed directory before trying to ./configure nginx.
• As an aside building pagespeed inside an alpine docker container is less than desirable (especially for any production purposes) unless you're going to copy the binary out and use just that in another container or you clean up the downloaded libraries (your container will be extremely bloated, which is the opposite of what you want with alpine).

[cchamplin]

Okay last night I put some time towards getting an APKBUILD put together for this. It should be available here https://github.com./cchamplin/alpine-ngx-pagespeed

This will need some testing as it's first draft, my guess is I might be missing a few makedepends.

This should also work a general instructions to get it ngx pagespeed built on Alpine.

FYI: This only works against nginx-1.9.11+ as it is linked as a dynamic module.

[iler]

@cchamplin Thanks a lot for your efforts! I will go through your repo and work from there.

And, yes, I'm aware of bloating the Docker container. I just wanted to slice the build to separate RUN commands to be able to use cache for the working parts of the build. If I encounter some problems I won't have to start downloading everything from scratch. Eventually when everything is working I will change this to one RUN command and do the proper cleanups. But thanks anyway :)

[ghost]

fyi the difference between compiling statically (without system libs) and with shared libraries (with system libs and deleted files) installing the required packages on alpine (apr apr-util libjpeg-turbo icu pcre) is ~4mb without even removing superfluous stuff added by the pkgs, so the packages themselves are pretty slim >.>

[iler]

Just to update here that I got this working. Here is my implementation for Docker https://github.com./iler/alpine-nginx-pagespeed if someone needs some advices on this. Thanks @cchamplin for the help!