Go's Defer
There are very few languages I’ve come across with as few “gotchas” as Go, for the most part, the language “just makes sense”. And for that reason, along with others, I am quick to say I really enjoy writing Go. And it is probably the language I write most often these days. However, there is one “gotcha” in particular that is really interesting, and that is defer.
defer is a great and powerful feature of Go. With it, you can ensure a function executes after the surrounding function finishes, regardless of whether it was successful or panicked. This is great for things like error recovery and cleaning up resources. But, this is a post about how defer has some “gotchas”, so…
There are some common “gotchas” with defer. Using defers in a loop, for example:
func example0() ([]byte, error) {
result := &bytes.Buffer{}
for i := 0; i < 10; i++ {
f, err := os.Open(
fmt.Sprintf("/some/file%d.txt", i),
)
if err != nil {
return nil, err
}
defer f.Close() // right here is the issue
data, err := ioutil.ReadAll(f)
if err != nil {
return nil, err
}
result.Write(append(data, '\n'))
}
return result.Bytes(), nil
}
Now the main issue is that defer only executes once the surounding function finishes. So within the for loop, f.Close() isn’t actually called until all files have been read and appended to the buffer. Granted, with only 10 files, this probably isn’t even an issue, but imagine doing this with thousands of files or network connections, then this becomes a real issue. Here is the same function, except now we are closing the file descriptor the moment we’re done with that particular file:
func example1() ([]byte, error) {
result := &bytes.Buffer{}
for i := 0; i < 10; i++ {
func(i int){
f, err := os.Open(
fmt.Sprintf("/some/file%d.txt", i),
)
if err != nil {
return nil, err
}
defer f.Close()
data, err := ioutil.ReadAll(f)
if err != nil {
return nil, err
}
result.Write(append(data, '\n'))
}(i)
}
return result.Bytes(), nil
}
So by wrapping the logic specific to an individual file in a self instantiating function, we open the file, read the file, concat the bytes, and then close the file descriptor before moving to the next file.
Another “gotcha”, that I’d also consider a feature, is the ability to modify return values, as such:
func example2() (i int) {
defer func(){
i = 3
}()
return 2
}
So to folks unfamiliar with how defer works, you’re probably expecting the return value to be 2, well if you run that function, you’ll actually get 3. The defer calls that function after the return statement is evaluated.
Now these “gotchas” are good to know, but these are just aspects of the language that are well documented and after a little while with the language you get used to these. But notice two phrases above, “only executes once the surrounding function finishes” and “the defer calls that function after the return statement is evaluated”. So based on these statements, it’d seem defer places the deferred function on the call stack and executes said deferred function after the previous function has returned, right? The Golang Tour even says “The deferred call’s arguments are evaluated immediately, but the function call is not executed until the surrounding function returns.”, As does the Go Blog. Well…
Take this example:
func example3() string {
defer example3()
fmt.Println("first")
return "second"
}
func main() {
fmt.Println(example3())
}
For one this is an infinite loop… But aside from that, if the surrounding function needs to return first, then shouldn’t this be a valid way to skirt around too many recursive calls? This stack overflow error would say otherwise:
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow
runtime stack:
runtime.throw(0x4a5f15, 0xe)
/usr/local/go/src/runtime/panic.go:596 +0x95
runtime.newstack(0x0)
/usr/local/go/src/runtime/stack.go:1089 +0x3f2
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:398 +0x86
goroutine 1 [running]:
fmt.(*fmt).fmt_s(0xc420426040, 0x4a5074, 0x5)
/usr/local/go/src/fmt/format.go:326 +0x70 fp=0xc440100358 sp=0xc440100350
fmt.(*pp).fmtString(0xc420426000, 0x4a5074, 0x5, 0x76)
/usr/local/go/src/fmt/print.go:430 +0x11f fp=0xc440100390 sp=0xc440100358
fmt.(*pp).printArg(0xc420426000, 0x489ca0, 0xc4202d9020, 0x76)
/usr/local/go/src/fmt/print.go:664 +0x7fc fp=0xc440100418 sp=0xc440100390
fmt.(*pp).doPrintln(0xc420426000, 0xc440100588, 0x1, 0x1)
/usr/local/go/src/fmt/print.go:1138 +0xa1 fp=0xc440100490 sp=0xc440100418
fmt.Fprintln(0x4f6160, 0xc42000c018, 0xc440100588, 0x1, 0x1, 0xc4202d9020, 0xc440100578, 0xc4202d9020)
/usr/local/go/src/fmt/print.go:247 +0x5c fp=0xc4401004f8 sp=0xc440100490
fmt.Println(0xc440100588, 0x1, 0x1, 0xc4202d9020, 0xc4401005a8, 0x10)
/usr/local/go/src/fmt/print.go:257 +0x57 fp=0xc440100548 sp=0xc4401004f8
main.ex(0x0, 0x0)
/home/christian/something.go:14 +0xec fp=0xc4401005a8 sp=0xc440100548
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100608 sp=0xc4401005a8
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100668 sp=0xc440100608
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc4401006c8 sp=0xc440100668
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100728 sp=0xc4401006c8
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100788 sp=0xc440100728
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc4401007e8 sp=0xc440100788
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100848 sp=0xc4401007e8
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc4401008a8 sp=0xc440100848
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100908 sp=0xc4401008a8
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc440100968 sp=0xc440100908
main.ex(0x4a5135, 0x6)
/home/christian/something.go:15 +0x102 fp=0xc4401009c8 sp=0xc440100968 // this line continues to repeat
main.ex(0x4a5135, 0x6)
Now when I first saw this, it was surprising, I had assumed that the defer function would execute once the surrounding function had returned, thus preventing the stack from growing out of control. However, as a printout of the assembly the go compiler indicates:
"".ex t=1 size=263 args=0x10 locals=0x58 [234/1847]
0x0000 00000 (something.go:7) TEXT "".ex(SB), $88-16
0x0000 00000 (something.go:7) MOVQ (TLS), CX
0x0009 00009 (something.go:7) CMPQ SP, 16(CX)
0x000d 00013 (something.go:7) JLS 253
0x0013 00019 (something.go:7) SUBQ $88, SP
0x0017 00023 (something.go:7) MOVQ BP, 80(SP)
0x001c 00028 (something.go:7) LEAQ 80(SP), BP
0x0021 00033 (something.go:7) FUNCDATA $0, gclocals·87f6052ef51eed84352c5a7cd7c29d63(SB)
0x0021 00033 (something.go:7) FUNCDATA $1, gclocals·488112b86ce2e5d099bbd25cfb5a88fa(SB)
0x0021 00033 (something.go:7) MOVQ $0, "".~r0+96(FP)
0x002a 00042 (something.go:7) MOVQ $0, "".~r0+104(FP)
0x0033 00051 (something.go:8) MOVL $16, (SP)
0x003a 00058 (something.go:8) LEAQ "".ex·f(SB), AX
0x0041 00065 (something.go:8) MOVQ AX, 8(SP)
0x0046 00070 (something.go:8) PCDATA $0, $1
0x0046 00070 (something.go:8) CALL runtime.deferproc(SB)
0x004b 00075 (something.go:8) TESTL AX, AX
0x004d 00077 (something.go:8) JNE 237
0x0053 00083 (something.go:9) LEAQ go.string."first"(SB), AX
0x005a 00090 (something.go:9) MOVQ AX, ""..autotmp_1+48(SP)
0x005f 00095 (something.go:9) MOVQ $5, ""..autotmp_1+56(SP)
0x0068 00104 (something.go:9) MOVQ $0, ""..autotmp_0+64(SP)
0x0071 00113 (something.go:9) MOVQ $0, ""..autotmp_0+72(SP)
0x007a 00122 (something.go:9) LEAQ type.string(SB), AX
0x0081 00129 (something.go:9) MOVQ AX, (SP)
0x0085 00133 (something.go:9) LEAQ ""..autotmp_1+48(SP), AX
0x008a 00138 (something.go:9) MOVQ AX, 8(SP)
0x008f 00143 (something.go:9) PCDATA $0, $2
0x008f 00143 (something.go:9) CALL runtime.convT2E(SB)
0x0094 00148 (something.go:9) MOVQ 24(SP), AX
0x0099 00153 (something.go:9) MOVQ 16(SP), CX
0x009e 00158 (something.go:9) MOVQ CX, ""..autotmp_0+64(SP)
0x00a3 00163 (something.go:9) MOVQ AX, ""..autotmp_0+72(SP)
0x00a8 00168 (something.go:9) LEAQ ""..autotmp_0+64(SP), AX
0x00ad 00173 (something.go:9) MOVQ AX, (SP)
0x00b1 00177 (something.go:9) MOVQ $1, 8(SP)
0x00ba 00186 (something.go:9) MOVQ $1, 16(SP)
0x00c3 00195 (something.go:9) PCDATA $0, $2
0x00c3 00195 (something.go:9) CALL fmt.Println(SB)
0x00c8 00200 (something.go:10) LEAQ go.string."second"(SB), AX
0x00cf 00207 (something.go:10) MOVQ AX, "".~r0+96(FP)
0x00d4 00212 (something.go:10) MOVQ $6, "".~r0+104(FP)
0x00dd 00221 (something.go:10) PCDATA $0, $1
0x00dd 00221 (something.go:10) XCHGL AX, AX
0x00de 00222 (something.go:10) CALL runtime.deferreturn(SB)
0x00e3 00227 (something.go:10) MOVQ 80(SP), BP
0x00e8 00232 (something.go:10) ADDQ $88, SP
0x00ec 00236 (something.go:10) RET
0x00ed 00237 (something.go:8) PCDATA $0, $1
0x00ed 00237 (something.go:8) XCHGL AX, AX
0x00ee 00238 (something.go:8) CALL runtime.deferreturn(SB)
0x00f3 00243 (something.go:8) MOVQ 80(SP), BP
0x00f8 00248 (something.go:8) ADDQ $88, SP
0x00fc 00252 (something.go:8) RET
0x00fd 00253 (something.go:8) NOP
0x00fd 00253 (something.go:7) PCDATA $0, $-1
0x00fd 00253 (something.go:7) CALL runtime.morestack_noctxt(SB)
0x0102 00258 (something.go:7) JMP 0
0x0000 64 48 8b 0c 25 00 00 00 00 48 3b 61 10 0f 86 ea dH..%....H;a....
0x0010 00 00 00 48 83 ec 58 48 89 6c 24 50 48 8d 6c 24 ...H..XH.l$PH.l$
0x0020 50 48 c7 44 24 60 00 00 00 00 48 c7 44 24 68 00 PH.D$`....H.D$h.
0x0030 00 00 00 c7 04 24 10 00 00 00 48 8d 05 00 00 00 .....$....H.....
0x0040 00 48 89 44 24 08 e8 00 00 00 00 85 c0 0f 85 9a .H.D$...........
0x0050 00 00 00 48 8d 05 00 00 00 00 48 89 44 24 30 48 ...H......H.D$0H
0x0060 c7 44 24 38 05 00 00 00 48 c7 44 24 40 00 00 00 .D$8....H.D$@...
0x0060 c7 44 24 38 05 00 00 00 48 c7 44 24 40 00 00 00 .D$8....H.D$@...
0x0070 00 48 c7 44 24 48 00 00 00 00 48 8d 05 00 00 00 .H.D$H....H.....
0x0080 00 48 89 04 24 48 8d 44 24 30 48 89 44 24 08 e8 .H..$H.D$0H.D$..
0x0090 00 00 00 00 48 8b 44 24 18 48 8b 4c 24 10 48 89 ....H.D$.H.L$.H.
0x00a0 4c 24 40 48 89 44 24 48 48 8d 44 24 40 48 89 04 L$@H.D$HH.D$@H..
0x00b0 24 48 c7 44 24 08 01 00 00 00 48 c7 44 24 10 01 $H.D$.....H.D$..
0x00c0 00 00 00 e8 00 00 00 00 48 8d 05 00 00 00 00 48 ........H......H
0x00d0 89 44 24 60 48 c7 44 24 68 06 00 00 00 90 e8 00 .D$`H.D$h.......
0x00e0 00 00 00 48 8b 6c 24 50 48 83 c4 58 c3 90 e8 00 ...H.l$PH..X....
0x00f0 00 00 00 48 8b 6c 24 50 48 83 c4 58 c3 e8 00 00 ...H.l$PH..X....
0x0100 00 00 e9 f9 fe ff ff .......
rel 5+4 t=16 TLS+0
rel 61+4 t=15 "".ex·f+0
rel 71+4 t=8 runtime.deferproc+0
rel 86+4 t=15 go.string."first"+0
rel 125+4 t=15 type.string+0
rel 144+4 t=8 runtime.convT2E+0
rel 196+4 t=8 fmt.Println+0
rel 203+4 t=15 go.string."second"+0
rel 223+4 t=8 runtime.deferreturn+0
rel 239+4 t=8 runtime.deferreturn+0
rel 254+4 t=8 runtime.morestack_noctxt+0
As we can see, the defer logic is called before the RET instruction, which would actually free the stack space. We can also see that the return statement itself for the surrounding function is called before the defer.
The take away I had when I first learned this was
1. Go functions do not return the same exact way C functions return.
2. Do not use defer as a hack to allow recursive calls since Go is not tail-recursion-optimized (shame on me :) ).
It is also worth mentioning Effective Go contradicts the prior two Go documentation links, and is actually the correct one.
So all in all, defer is a powerful feature of Go, but it is tricky to say the least.
Until next time, later folks!