Saturday, April 26, 2014

Calling Conventions

Calling conventions is very important concept of C++ programming. Calling conventions decides two things, how arguments will be passed on stack and who will clear the stack.

In this article I will describe (and will prove it) two calling conventions, __cdecl and __stdcall.

In case of both __cdecl and __stdcall, arguments will be pushed on stack from right to left. Difference is, in __cdecl caller clears the stack, while in __stdcall callee will clear stack.

Let’s consider following code,

int __stdcall StdCall(int a,int b)
      return a + b;

int __cdecl Cdecl(int a,int b)
      return a + b;

int main(int argc, char* argv[])
      return 0;

In this code, StdCall and Cdecl take two ints and returns there addition. I kept code simple to illustrate concept.
Now we will see code’s assembly equivalent to understand how calling conventions do the trick.  I am using Visual Studio to get assembly code. Here I am showing assembly code in parts.

Call to StdCall

push  20                            ; 00000014H
push  10                            ; 0000000aH
call  ?StdCall@@YGHHH@Z             ; StdCall

 Definition of StdCall

?StdCall@@YGHHH@Z PROC                                 ; StdCall, COMDAT
       push   ebp
       mov    ebp, esp
       sub    esp, 192                          ; 000000c0H
       push   ebx
       push   esi
       push   edi
       lea    edi, DWORD PTR [ebp-192]
       mov    ecx, 48                                  ; 00000030H
       mov    eax, -858993460                          ; ccccccccH
       rep stosd
       mov    eax, DWORD PTR _a$[ebp]
       add    eax, DWORD PTR _b$[ebp]
       pop    edi
       pop    esi
       pop    ebx
       mov    esp, ebp
       pop    ebp
       ret    8
?StdCall@@YGHHH@Z ENDP                                 ; StdCall

In call to StdCall, it pushes arguments from right to left, i.e. 20, 10. And look at definition of StdCall, you can see ret 8. This is return statement, and don’t get confused with return statement of C++, here return statement doesn’t return value, but pass control from where this function getting called. ret 8 means it will return control by clearing stack with 8 bytes (adds 8 bytes to stack pointer, esp ). In 32 bit OS int are of 4 bytes, we are passing two ints so that will make it 8 bytes. So callee has cleaned the stack.

Now look at Cdecl calling convention.

Call to Cdecl

push  20                            ; 00000014H
push  10                            ; 0000000aH
call  ?Cdecl@@YAHHH@Z                     ; Cdecl
add   esp, 8

Definition of Cdecl

?Cdecl@@YAHHH@Z PROC                            ; Cdecl, COMDAT
      push  ebp
      mov   ebp, esp
      sub   esp, 192                      ; 000000c0H
      push  ebx
      push  esi
      push  edi
      lea   edi, DWORD PTR [ebp-192]
      mov   ecx, 48                             ; 00000030H
      mov   eax, -858993460                     ; ccccccccH
      rep stosd
      mov   eax, DWORD PTR _a$[ebp]
      add   eax, DWORD PTR _b$[ebp]
      pop   edi
      pop   esi
      pop   ebx
      mov   esp, ebp
      pop   ebp
      ret   0
?Cdecl@@YAHHH@Z ENDP                            ; Cdecl

Like StdCall, here in Cdecl both the arguments are pushed from right to left, and then call to function is made. However after call statement, you can see              add  esp, 8, which adds 8 to esp. esp points to top of stack. Adding 8 to means it is clearing 8 bytes (because stack grows from higher to lower memory). So here caller is clearing stack. Contrary to StdCall here you can see, ret 0, which just returns control and do not anything to stack pointer. So, in this case caller clears the stack.

Please add comments for your queries and suggestions. 

1 comment: