gitea/tea
gitea/tea
57
135
Fork 67
Code Issues 102 Pull Requests 8 Actions Releases 16 Activity

Makefile: add STATIC=true for static PIE builds #349

Merged
6543 merged 13 commits from noerw/tea:make-static into master 2021-09-23 16:01:07 +00:00
Conversation 5 Commits 13 Files Changed 4 +27 -31
noerw commented 2021-03-15 14:51:56 +00:00
Member
Copy Link
  • make build + make install now support the STATIC=true parameter, creating statically linked builds that are also position independent executables
    • this requires CGO and a static libc on the build system
  • CGO_ENABLED=0 is set for all make build targets, unless STATIC=true is set
  • Debug symbols are stripped (-s -w) for all make build targets
  • Release binaries are built statically by gox (no PIE), as before.

I also took the liberty to declutter the makefile from unused & duplicated variables.

Tested on linux-amd64 (fedora, arch, alpine with go 1.15.8 + 1.16.2), as well as windows 10.

- `make build` + `make install` now support the `STATIC=true` parameter, creating statically linked builds that are also position independent executables - this requires CGO and a static libc on the build system - `CGO_ENABLED=0` is set for all make build targets, unless `STATIC=true` is set - Debug symbols are stripped (`-s -w`) for all make build targets - Release binaries are built statically by gox (no PIE), as before. I also took the liberty to declutter the makefile from unused & duplicated variables. Tested on linux-amd64 (fedora, arch, alpine with go 1.15.8 + 1.16.2), as well as windows 10.
noerw added 1 commit 2021-03-15 14:51:57 +00:00
allow static builds via make STATIC=true
All checks were successful
continuous-integration/drone/pr Build is passing
Details
a323a8d1fd
noerw added the
kind/build
 label 2021-03-15 14:54:53 +00:00
noerw added the
status/wip
 label 2021-03-15 14:56:44 +00:00
noerw commented 2021-03-15 18:38:12 +00:00
Author
Member
Copy Link

ok, on alpine ldd says there's still a reference to ld-musl, not sure if there's a way around that

bash-5.1# cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.13.2
PRETTY_NAME="Alpine Linux v3.13"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

bash-5.1# make build STATIC=true
go build -v -mod=vendor -tags '' -ldflags '-X "main.Version=" -X "main.Tags="' -tags 'osusergo,netgo,static_build,' -ldflags '-X "main.Version=" -X "main.Tags=" -extldflags "-fno-PIC -static" -s -w -s -w' -buildmode=pie -o tea

bash-5.1# ldd ./tea
        /lib/ld-musl-x86_64.so.1 (0x7fc7c212a000)

bash-5.1# make build STATIC=false
go build -v -mod=vendor -tags '' -ldflags '-X "main.Version=" -X "main.Tags="' -o tea

bash-5.1# ldd ./tea
        /lib/ld-musl-x86_64.so.1 (0x7faa982ba000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7faa982ba000)
ok, on alpine `ldd` says there's still a reference to ld-musl, not sure if there's a way around that ``` bash-5.1# cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.13.2 PRETTY_NAME="Alpine Linux v3.13" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://bugs.alpinelinux.org/" bash-5.1# make build STATIC=true go build -v -mod=vendor -tags '' -ldflags '-X "main.Version=" -X "main.Tags="' -tags 'osusergo,netgo,static_build,' -ldflags '-X "main.Version=" -X "main.Tags=" -extldflags "-fno-PIC -static" -s -w -s -w' -buildmode=pie -o tea bash-5.1# ldd ./tea /lib/ld-musl-x86_64.so.1 (0x7fc7c212a000) bash-5.1# make build STATIC=false go build -v -mod=vendor -tags '' -ldflags '-X "main.Version=" -X "main.Tags="' -o tea bash-5.1# ldd ./tea /lib/ld-musl-x86_64.so.1 (0x7faa982ba000) libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7faa982ba000) ```
noerw added 2 commits 2021-03-17 11:54:21 +00:00 
remove unused / duplicate variables
make: fix STATIC=true
All checks were successful
continuous-integration/drone/pr Build is passing
Details
6c219abc6c 
- -buildmode=pie silently creates dynamically linked executables
- also was passing the wrong tags to main.Tags
noerw commented 2021-03-17 12:12:41 +00:00
Author
Member
Copy Link

Ok, so this works now quite well.

  • make build + make install now support the STATIC=true parameter, creating statically linked builds, instead of dynamically linked PIE executables
    • a limitation of the go toolchain is that -buildmode=pie silently switches to generate dynamic executables, so it's not feasible to produce a static PIE exe. Supposedly there is a way around that but I didn't get this to work across platforms.
  • CGO_ENABLED=0 is set for all make build targets, which seems a reasonable default for tea
  • Debug symbols are stripped (-s -w) for all make build targets
  • Release binaries are build statically by gox, as before.

I also took the liberty to declutter the makefile from unused & duplicated variables, let me know if I dropped something important (MAKE_VERSION?)

I tested on linux-amd64 (fedora, arch, alpine with go 1.15 + 1.16), as well as windows 10.

Ok, so this works now quite well. - `make build` + `make install` now support the `STATIC=true` parameter, creating statically linked builds, instead of dynamically linked PIE executables - a limitation of the go toolchain is that `-buildmode=pie` silently switches to generate dynamic executables, so it's not feasible to produce a static PIE exe. [Supposedly there is a way around that](https://github.com/golang/go/issues/40711#issuecomment-750517228) but I didn't get this to work across platforms. - `CGO_ENABLED=0` is set for all make build targets, which seems a reasonable default for `tea` - Debug symbols are stripped (`-s -w`) for all make build targets - Release binaries are build statically by gox, as before. I also took the liberty to declutter the makefile from unused & duplicated variables, let me know if I dropped something important (`MAKE_VERSION`?) I tested on linux-amd64 (fedora, arch, alpine with go 1.15 + 1.16), as well as windows 10.
🎉 1
noerw removed the
status/wip
 label 2021-03-17 12:13:06 +00:00
noerw added 1 commit 2021-04-08 12:43:31 +00:00
zeripath reviewed 2021-05-10 13:48:21 +00:00
Makefile Outdated
@ -43,1 +26,3 @@
LDFLAGS := -X "main.Version=$(TEA_VERSION)" -X "main.Tags=$(TAGS)"
TAGS ?=
TAGS_STATIC := osusergo,netgo,static_build,$(TAGS)
zeripath commented 2021-05-10 13:42:54 +00:00
Owner
Copy Link

does anything actually use "static_build" as a build tag? There's nothing in go's source that uses that tag and nothing in gitea's vendor tree. Does tea have a dependency that uses that tag - or is this a futureproof thing?

does anything actually use "static_build" as a build tag? There's nothing in go's source that uses that tag and nothing in gitea's vendor tree. Does tea have a dependency that uses that tag - or is this a futureproof thing?
noerw commented 2021-05-10 14:38:39 +00:00
Author
Member
Copy Link

I added this as a safety, my only reference is this though. We can drop it or replace it with static as suggested in that thread

I added this as a safety, my only reference is [this](https://github.com/golang/go/issues/26492#issuecomment-407336850) though. We can drop it or replace it with `static` as suggested in that thread
zeripath commented 2021-05-10 20:20:55 +00:00
Owner
Copy Link

no it's fine but perhaps just add a comment in the makefile referencing why you've added it - so that we know where it has come from in case it causes trouble.

no it's fine but perhaps just add a comment in the makefile referencing why you've added it - so that we know where it has come from in case it causes trouble.
noerw marked this conversation as resolved
Makefile Outdated
@ -44,0 +27,4 @@
TAGS ?=
TAGS_STATIC := osusergo,netgo,static_build,$(TAGS)
LDFLAGS := -X "main.Version=$(TEA_VERSION)" -X "main.Tags=$(TAGS)" -s -w
zeripath commented 2021-05-10 13:48:04 +00:00
Owner
Copy Link

It's usually nicer to have something like:

LDFLAGS := $(LDFLAGS) -X "main.Version=$(TEA_VERSION)" -X "main.Tags=$(TAGS)" -s -w

as that allows people to adjust and add to the LDFLAGS at make time.

It's usually nicer to have something like: ``` LDFLAGS := $(LDFLAGS) -X "main.Version=$(TEA_VERSION)" -X "main.Tags=$(TAGS)" -s -w ``` as that allows people to adjust and add to the LDFLAGS at make time.
noerw commented 2021-05-10 15:44:41 +00:00
Author
Member
Copy Link

good point, will add that

good point, will add that
noerw commented 2021-05-10 20:39:57 +00:00
Author
Member
Copy Link

I managed to add this for GOFLAGS, but for LDFLAGS I didn't find a syntax that didn't either

  • reset the variable completely, when values are specified on the CLI
  • ignore overrides from the STATIC == true if clause

If you know how, I'm happy to add that

I managed to add this for `GOFLAGS`, but for `LDFLAGS` I didn't find a syntax that didn't either - reset the variable completely, when values are specified on the CLI - ignore overrides from the STATIC == true if clause If you know how, I'm happy to add that
Makefile Outdated
@ -44,0 +32,4 @@
# TODO: clean this mess up when there are news on https://github.com/golang/go/issues/26492
# note: -buildmode=pie is incompatible with static builds as of go 1.16!
LDFLAGS_STATIC := $(LDFLAGS) -extldflags "-fno-PIC -static" -X "main.Tags=$(TAGS_STATIC)"
zeripath commented 2021-05-10 13:45:52 +00:00
Owner
Copy Link

it's interesting that you've included the "-fno-PIC -static" extldflags but I see you have CGO_ENABLED=0 so I don't think these will ever actually get called.

it's interesting that you've included the "-fno-PIC -static" extldflags but I see you have CGO_ENABLED=0 so I don't think these will ever actually get called.
noerw commented 2021-05-10 15:45:12 +00:00
Author
Member
Copy Link

You're right, -extldflags isn't used due to missing -linkmode=external. Will check how to resolve that.
But your assumption is wrong; -linkmode=external works with CGO_ENABLED=0

edit: actually under some conditions go build implicilty sets linkmode=external, so these options are used after all.

You're right, `-extldflags` isn't used due to missing `-linkmode=external`. Will check how to resolve that. But your assumption is wrong; `-linkmode=external` works with `CGO_ENABLED=0` edit: actually under some conditions `go build` implicilty sets linkmode=external, so these options are used after all.
zeripath commented 2021-05-10 20:21:36 +00:00
Owner
Copy Link

Cool thanks for checking.

Cool thanks for checking.
zeripath marked this conversation as resolved
noerw added 3 commits 2021-05-10 20:37:25 +00:00 
this is documented nowhere, but I finally found a way to have both of
- static builds
- position independent executables

exotic flags of arcane build tools ftw!
update dockerfile to work with make STATIC=true
All checks were successful
continuous-integration/drone/pr Build is passing
Details
da5898c4aa 
and add a .dockerignore
techknowlogick reviewed 2021-05-10 20:41:39 +00:00
Makefile Outdated
@ -158,3 +151,3 @@
cd /tmp && $(GO) get -u github.com/mitchellh/gox; \
fi
CGO_ENABLED=0 gox -verbose -cgo=false -tags '$(TAGS)' -ldflags '-s -w $(LDFLAGS)' -osarch='!darwin/386 !darwin/arm64 !darwin/arm' -os="windows linux darwin" -arch="386 amd64 arm arm64" -output="$(DIST)/release/tea-$(VERSION)-{{.OS}}-{{.Arch}}"
CGO_ENABLED=0 gox -verbose -cgo=false $(GOFLAGS_STATIC) -osarch='!darwin/386 !darwin/arm64 !darwin/arm' -os="windows linux darwin" -arch="386 amd64 arm arm64" -output="$(DIST)/release/tea-$(VERSION)-{{.OS}}-{{.Arch}}"
techknowlogick commented 2021-05-10 20:41:39 +00:00
Owner
Copy Link

Should/could darwin/arm64 be added in (either via this PR or another)?

Should/could darwin/arm64 be added in (either via this PR or another)?
noerw commented 2021-05-10 21:03:23 +00:00
Author
Member
Copy Link

I created another PR #360

I created another PR https://gitea.com/gitea/tea/pulls/360
noerw marked this conversation as resolved
noerw added 1 commit 2021-05-10 20:50:34 +00:00
fix make release
All checks were successful
continuous-integration/drone/pr Build is passing
Details
ec58ba91bf
noerw commented 2021-05-10 20:53:23 +00:00
Author
Member
Copy Link

I finally found a way to build a PIE that is statically linked 🎉 (quite an achievement, this isn't documented anywhere)

Compared to the previous solution, there are no more caveats building statically, other than

  • need to install glibc-static or equivalent
  • resulting binary is 25M (make STATIC=true) instead of 18M (CGO_ENABLED=0 go build -ldflags '-s -w' .)

This is ready from my side now

I finally found a way to build a PIE that is statically linked :tada: (quite an achievement, this isn't documented *anywhere*) Compared to the previous solution, there are no more caveats building statically, other than - need to install `glibc-static` or equivalent - resulting binary is 25M (`make STATIC=true`) instead of 18M (`CGO_ENABLED=0 go build -ldflags '-s -w' .`) This is ready from my side now
noerw added 1 commit 2021-05-10 20:57:32 +00:00
document static_build tag
All checks were successful
continuous-integration/drone/pr Build is passing
Details
b0521a7391
noerw changed title from Allow static builds to Makefile: add STATIC=true for static PIE builds 2021-05-11 10:02:19 +00:00
zeripath approved these changes 2021-06-22 18:02:25 +00:00
lunny commented 2021-06-23 05:29:39 +00:00
Owner
Copy Link

Please resolve the conflicts.

Please resolve the conflicts.
noerw added 1 commit 2021-06-24 10:27:17 +00:00
Merge branch 'master' into make-static
All checks were successful
continuous-integration/drone/pr Build is passing
Details
38168aab67
noerw added 1 commit 2021-06-29 10:19:30 +00:00
6543 added 1 commit 2021-08-12 15:03:12 +00:00
Merge branch 'master' into make-static
All checks were successful
continuous-integration/drone/pr Build is passing
Details
ce624ca25f
6543 reviewed 2021-08-12 15:20:21 +00:00
README.md Outdated
@ -101,3 +101,3 @@
git clone https://gitea.com/gitea/tea.git
cd tea
make
make STATIC=true
6543 commented 2021-08-12 15:20:21 +00:00
Owner
Copy Link

not STATIC=true make ?

not `STATIC=true make` ?
6543 commented 2021-08-12 15:23:38 +00:00
Owner
Copy Link
> STATIC=false make clean build

> ldd tea
statically linked

static linekd disabled do still link statically & if enabled I get following compile issue:

go build -buildmode=pie  -mod=vendor -tags 'osusergo,netgo,static_build,' -ldflags '-X "main.Version=0.7.0+25-gce624ca" -X "main.Tags=" -s -w -linkmode=external -extldflags "-static-pie" -X "main.Tags=osusergo,netgo,static_build,"' -o tea
# code.gitea.io/tea
loadinternal: cannot find runtime/cgo
``` > STATIC=false make clean build > ldd tea statically linked ``` static linekd disabled do still link statically & if enabled I get following compile issue: ``` go build -buildmode=pie -mod=vendor -tags 'osusergo,netgo,static_build,' -ldflags '-X "main.Version=0.7.0+25-gce624ca" -X "main.Tags=" -s -w -linkmode=external -extldflags "-static-pie" -X "main.Tags=osusergo,netgo,static_build,"' -o tea # code.gitea.io/tea loadinternal: cannot find runtime/cgo ```
noerw added 1 commit 2021-09-23 11:26:31 +00:00
Merge branch 'master' into make-static
All checks were successful
continuous-integration/drone/pr Build is passing
Details
46fe9b0075
6543 approved these changes 2021-09-23 15:59:44 +00:00
6543 merged commit 555f1ae516 into master 2021-09-23 16:01:07 +00:00
noerw deleted branch make-static 2021-10-24 13:22:24 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
zeripath
6543
Labels
Clear labels   
kind/breaking

   
kind/bug

   
kind/build

   
kind/dependency

   
kind/deployment

   
kind/docs
  
kind
enhancement

  
kind
feature

  
kind/proposal
  
kind
question

  
kind
refactor

  
kind/security

   
kind/testing

   
kind/translation

   
priority/critical

   
priority/high

   
priority/low

   
priority/medium

   
reviewed/duplicate

   
reviewed/invalid

   
reviewed/wontfix

   
skip-changelog

   
status/blocked

   
status/has-backport

   
status/has-pull

   
status/needs-backport

   
status/needs-feedback

   
status/needs-reviews

   
status/wip

   
upstream/gitea

   
upstream/sdk
No Label
kind/breaking
kind/bug
kind/build
kind/dependency
kind/deployment
kind/docs
kind
enhancement
kind
feature
kind/proposal
kind
question
kind
refactor
kind/security
kind/testing
kind/translation
priority/critical
priority/high
priority/low
priority/medium
reviewed/duplicate
reviewed/invalid
reviewed/wontfix
skip-changelog
status/blocked
status/has-backport
status/has-pull
status/needs-backport
status/needs-feedback
status/needs-reviews
status/wip
upstream/gitea
upstream/sdk
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
5 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: gitea/tea#349
No description provided.
Delete Branch "noerw/tea:make-static"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?