core22 min studyLesson 104 in Engineering

Calling conventions: make functions agree

Trace System V AMD64 argument registers, preserved state, return addresses, and ordinary stack alignment.

On this page

What you’ll understand

  • Identify caller-saved and callee-saved responsibilities in System V AMD64.
  • Calculate ordinary 16-byte stack alignment across a call.
  • Explain why stack frames and frame pointers are conventions with optimization-dependent shapes.

A call instruction transfers control, but separately compiled functions need a wider agreement: where arguments arrive, where a result goes, and which state survives. An application binary interface supplies that agreement. This lesson uses the ordinary System V AMD64 convention for integer arguments and calls without stack-passed vector arguments. Other platforms, including Windows x64, follow different rules; compatible instruction encodings do not make their calling conventions interchangeable.

Arguments are assigned by a shared convention

For ordinary integer or pointer arguments in the System V AMD64 register classes, the first six argument registers are RDI, RSI, RDX, RCX, R8, and R9. A simple integer return value commonly uses RAX. Classification rules become more involved for aggregates, floating-point values, variadic functions, and wide vectors, so this six-register list is a deliberately bounded starting point. A compiler knows those classifications and emits code that lets independently compiled caller and callee agree without sharing local variable names.

Caller-saved registers may be changed by the callee; a caller that needs their values afterward must arrange preservation. Callee-saved RBX, RBP, and R12 through R15 must be restored if used, and RSP must be correctly restored for returning. This division is an ownership rule rather than a promise that every function saves every register. A leaf function that changes none of the preserved registers can avoid those saves. Inspect actual liveness before attributing every register move to the ABI.

Nstack=max⁡(0,Ninteger−6)N_{\rm stack}=\max(0,N_{\rm integer}-6)
For a simplified call containing only ordinary one-register integer-class arguments, this counts arguments beyond the six available registers. Real ABI classification, alignment, and aggregate rules can change stack placement; the count is not a universal call-layout formula.↔ Scroll wide equations horizontally.

Alignment is a relationship across the call

For an ordinary call with the base 16-byte stack alignment requirement, the caller arranges RSP to be divisible by 16 immediately before CALL. CALL pushes an eight-byte return address, so the callee’s entry RSP is eight modulo 16. Equivalently, RSP plus eight is aligned at entry. Saving one eight-byte register changes the remainder again. Stronger alignment requirements apply for some stack-passed vector arguments, which this example excludes. Work in bytes and check the boundary at which the ABI states its rule.

A stack frame is storage associated with an active call, not necessarily a fixed picture with every field present. A compiler may omit a dedicated frame pointer, keep locals in registers, inline a call, or perform a tail call that reuses the caller’s return path. Debugging and unwinding can use metadata rather than a linked list of RBP frames. The user-space red zone also has specific ABI conditions; do not copy assumptions about it into kernel code or a different calling convention.

(RSPbefore call) mod 16=0,(RSPentry+8) mod 16=0(RSP_{\rm before\ call})\bmod16=0,\qquad (RSP_{\rm entry}+8)\bmod16=0
RSP values are byte addresses for the stated ordinary System V AMD64 case. CALL contributes an eight-byte return address. This excludes calls requiring stronger stack alignment for particular stack arguments.↔ Scroll wide equations horizontally.

Preserve the contract while changing the implementation

Handwritten assembly, foreign-function interfaces, and JIT-generated code must obey the same visible calling contract as compiler-generated functions. A function that produces the expected result but corrupts RBX can fail much later when its caller reuses that register. Likewise, a stack-alignment violation can remain hidden until a callee performs an operation with stricter assumptions. Test boundaries with representative signatures and preserved-register checks, not only a trivial call that happens to return the right integer.

The network lab exposes a single sigmoid computation with input x and weight w. It is useful as an analogy for an interface: predict the output from the documented inputs, then change one input while holding the other fixed. The diagram does not represent argument registers, a call stack, or a trainable network. The implementation-independent idea is that a caller can reason from a stable contract. Use a separate register-and-stack trace when applying that idea to an actual binary interface.

Align a nested ordinary call

A function enters with RSP=0x1008. It pushes RBX, then wants 24 bytes of local payload before making another ordinary System V AMD64 call. No stronger vector alignment is required.

  1. At entry RSP is eight modulo 16, as required after a caller’s eight-byte return address push. Pushing RBX subtracts eight bytes, leaving RSP=0x1000, now divisible by 16.

  2. Subtracting only 24 bytes would leave 0x0FE8, which is eight modulo 16 and unsuitable immediately before the nested ordinary call. Round the local allocation to 32 bytes, yielding RSP=0x0FE0.

  3. The nested CALL pushes its return address, so its callee enters at 0x0FD8, eight modulo 16. After that callee returns, release the 32-byte allocation and restore RBX before this function returns.

The result

The local payload needs eight bytes of padding in this layout. Alignment is checked before the nested call and must be balanced on return.

Explore the model

Interactive lab

Interactive lab · network

Inputs and an inspectable interface

Change input x and weight w in the fixed computation σ(wx−0.5). Predict the output direction before moving a control. This is an interface analogy; it does not model ABI registers, stack frames, or function calls.

Inputs and an inspectable interfaceOne inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Current readings: Product · wx: 1.2; Logit · wx − 0.5: 0.7; Sigmoid output: 0.6682. All parameters are adjustable with the labeled controls below.One input. One weight. One sigmoid.x1.20× w1.20+ b0.70σ0.668output0.668w = 1.00b = −0.50σ(z) = 1 / (1 + exp(−z))
Computation graphSignal trace
Product · wx
1.2
Logit · wx − 0.5
0.7
Sigmoid output
0.6682
t = 0 s

One inspectable sigmoid unit, σ(wx − 0.5). Playback traces the computation graph. There is no training, attention mechanism or hidden deep network. Parameter changes pause playback and recompute the model at the current elapsed time. Reset restores both parameters and returns time to zero. Playback starts paused. Adjust either parameter, then choose Play or Step.

Look forA stable input-output contract supports reasoning even when the internal implementation changes.

Check your understanding

A System V AMD64 function uses RBX as a temporary. What must it ensure before returning normally?

Try it yourself

A callee enters with RSP=0x2008, saves no registers, and needs 16 bytes of local payload before another ordinary call. What allocation satisfies base alignment?

Take a moment to make a prediction or work it out on paper. Open a hint when you need one.

Give me a hint

Entry RSP is eight modulo 16; subtract an amount that makes the pre-call value zero modulo 16.

Compare with the solution

Allocate 24 bytes: RSP becomes 0x1FF0, divisible by 16, with 16 bytes of payload and eight bytes of padding. A 16-byte allocation alone would leave RSP eight modulo 16. Restore the full 24 bytes before returning, assuming no stronger alignment requirement.

Take it with you

  • An ABI defines shared register, stack, and data-layout responsibilities.
  • Caller-saved and callee-saved state divide preservation work.
  • Check alignment at the documented call boundary and restore balanced state.

Read further

Original sources and deeper explanations to follow at your own pace.

Optional progress, saved in this browser when available. Revisit any lesson at any time.

Lesson not marked complete.