Should I use DT_UNSBIGINT for HKEY values returned from a 64-bit external procedure call?

[Breck_Carter]

Overview: The short answer is "Yes" (Martin answered, John Smirnios confirmed, and testing proved it). HKEY values are registry handles which are actually 32-bit pointers in 32-bit Windows and SQL Anywhere, and 64-bit pointers in 64-bit Windows and SQL Anywhere. When passing HKEY values between SQL and C via the external call interface, there is no automatic type compensation process, so for 32-bits the HKEY must be declared as UNSIGNED INTEGER in the SQL CREATE PROCEDURE, HKEY in the 32-bit C dll, and DT_UNSINT when calling set_value, whereas for 64-bits it must be declared as UNSIGNED BIGINT in the SQL CREATE PROCEDURE, HKEY in the 64-bit C dll, and DT_UNSBIGINT when calling set_value.

---

Update: The full code story has been included (scroll down), and the links have been fixed.

Also: The symptom I am getting is that a call to RegEnumValue inside a C external procedure is setting the return code to 6 which means ERROR_INVALID_HANDLE.

---

Here are some snippets of code from a 32-bit C dll that is functioning correctly when called from SQL using the "classic" SQL Anywhere external call interface.

In this code, the values in odbc_hkey and dsn_index are returned to the caller:

__declspec(dllexport) _VOID_ENTRY get_first_dsn ( an_extfn_api *api, void *arg_handle ) {
...
   an_extfn_value  api_odbc_hkey;
   an_extfn_value  api_dsn_index;
...
   HKEY            odbc_hkey; 
   DWORD           dsn_index;
...
   api_odbc_hkey.type = DT_INT;
   api_odbc_hkey.data = &odbc_hkey;
   api -> set_value ( arg_handle, 2, &api_odbc_hkey, FALSE );

api_dsn_index.type = DT_INT;
   api_dsn_index.data = &dsn_index;
   api -> set_value ( arg_handle, 3, &api_dsn_index, FALSE );

My questions...

(a) Is it true that I should have been using DT_UNSINT for the DWORD thing, but it doesn't really matter when push comes to shove?

(b) Is it true that DT_UNSINT will work with DWORD even if I create a 64-bit version of the C dll? And that corresponds to UNSIGNED INT on the SQL side?

(c) Is it true that I should have been using DT_UNSINT for the HKEY thing in the 32-bit version, but again, it doesn't really matter?

(d) Is it true, however, that using DT_INT or DT_UNSINT with HKEY is VERY WRONG for a 64-bit version of the dll... that I MUST use DT_UNSBIGINT? ...and in the SQL, UNSIGNED BIGINT?

(e) And that might explain why I'm getting the occasional "bad handle" errors with the 64-bit version?

---

Here's my research...

(1) a C routine wants to pass a DWORD back and forth to the calling SQL code:

Windows Data Types

DWORD 

A 32-bit unsigned integer. The range is 0 through 4294967295 decimal.
This type is declared in WinDef.h as follows:
typedef unsigned long DWORD;

(2) in "Embedded SQL-Speak" that is a DT_UNSINT:

Embedded SQL data types

DT_UNSINT   32-bit unsigned integer.

(3) which means that UNSIGNED INT should be used in the calling SQL code:

Handling data types

The following SQL data types can be passed to an external library:
SQL data type      sqldef.h            C type
[ UNSIGNED ] INT   DT_INT, DT_UNSINT   [ Unsigned ] 4-byte integer

---

Now, what about this case:

(4) a C routine wants to pass an HKEY back and forth to the calling SQL code, but the trail runs cold at this circular definition: "typedef void *PVOID"... what is the actual base data type?

Windows Data Types

HKEY  

A handle to a registry key.
This type is declared in WinDef.h as follows:
typedef HANDLE HKEY;
HANDLE

A handle to an object.
This type is declared in WinNT.h as follows:
typedef PVOID HANDLE;
PVOID 

A pointer to any type.
This type is declared in WinNT.h as follows:
typedef void *PVOID;

(5) I am guessing I want to use DT_UNSINT for the 32-bit version of the C routine, and DT_UNSBIGINT in the 64-bit version:

Embedded SQL data types

DT_UNSINT      32-bit unsigned integer
DT_UNSBIGINT   64-bit unsigned integer.

(6) which means that back in the SQL code I want to use UNSIGNED INT when passing an HKEY back and forth to the 32-bit C routine, and UNSIGNED BIGINT when calling the 64-bit C routine:

Handling data types

The following SQL data types can be passed to an external library:
SQL data type          sqldef.h                  C type
[ UNSIGNED ] INT       DT_INT, DT_UNSINT         [ Unsigned ] 4-byte integer
[ UNSIGNED ] BIGINT    DT_BIGINT, DT_UNSBIGINT   [ Unsigned ] 8-byte integer

Izzat right?

---

Here's the full code story, starting with the CREATE PROCEDURE EXTERNAL definitions, followed by the C code for a pair of C functions "get first" and "get next".

The problem is that the call to RegEnumValue in the first call to rroad_get_next_dsn / get_next_dsn is setting the return code to 6 which means ERROR_INVALID_HANDLE:

DECLARE @sql                          LONG VARCHAR;

SET @sql = STRING ( 
   'CREATE PROCEDURE rroad_get_first_dsn ( ',
      'IN    dsn_type_code     INTEGER, ',
      'OUT   odbc_hkey         INTEGER, ',
      'OUT   dsn_index         INTEGER, ',
      'OUT   odbc_dsn          VARCHAR ( 255 ), ',
      'OUT   odbc_driver       VARCHAR ( 255 ), ',
      'OUT   return_code       INTEGER, ',
      'OUT   diagnostic_code   INTEGER, ',
      'OUT   diagnostic_string VARCHAR ( 255 ) ) ',
   'EXTERNAL NAME ''get_first_dsn@', @foxhound_db_folder, '\\\\', @rroad1_dll_name, '''' );

EXECUTE IMMEDIATE @sql;

SET @sql = STRING ( 
   'CREATE PROCEDURE rroad_get_next_dsn ( ',
      'IN    odbc_hkey         INTEGER, ',
      'INOUT dsn_index         INTEGER, ',
      'OUT   odbc_dsn          VARCHAR ( 255 ), ',
      'OUT   odbc_driver       VARCHAR ( 255 ), ',
      'OUT   return_code       INTEGER, ',
      'OUT   diagnostic_code   INTEGER, ',
      'OUT   diagnostic_string VARCHAR ( 255 ) ) ',
   'EXTERNAL NAME ''get_next_dsn@', @foxhound_db_folder, '\\\\', @rroad1_dll_name, '''' );

EXECUTE IMMEDIATE @sql;

//------------------------------------------------------------------------------------------------
// get_first_dsn
//------------------------------------------------------------------------------------------------

__declspec(dllexport) _VOID_ENTRY get_first_dsn ( an_extfn_api *api, void *arg_handle ) {

an_extfn_value  api_dsn_type_code;
   an_extfn_value  api_odbc_hkey;
   an_extfn_value  api_dsn_index;
   an_extfn_value  api_odbc_dsn;
   an_extfn_value  api_odbc_driver;
   an_extfn_value  api_return_code;
   an_extfn_value  api_diagnostic_code;
   an_extfn_value  api_diagnostic_string;

DWORD           dsn_type_code;
   HKEY            odbc_hkey; 
   DWORD           dsn_index;

char *          odbc_dsn; 
   DWORD           odbc_dsn_size;

char *          odbc_driver; 
   DWORD           odbc_driver_size;

DWORD           return_code; 
   DWORD           diagnostic_code; 
   char *          diagnostic_string;

if( !api -> get_value ( arg_handle, 1, &api_dsn_type_code ) || api_dsn_type_code.data == NULL ) {
      diagnostic_code = 1;
      return;
   }

dsn_type_code = *(DWORD *) api_dsn_type_code.data;

odbc_hkey = 0;
   dsn_index = 0;

odbc_dsn  = (char *) malloc( 255 );
   strcpy_s ( odbc_dsn, 255, "" );
   odbc_dsn_size = 255;

odbc_driver = (char *) malloc( 255 );
   strcpy_s ( odbc_driver, 255, "" );
   odbc_driver_size = 255;

return_code        = 0;     
   diagnostic_code    = 901;     
   diagnostic_string  = (char *) malloc( 255 );
   strcpy_s ( diagnostic_string, 255, "//-901" );

// Open ODBC.INI\\ODBC Data Sources key.

if ( dsn_type_code == 1 ) {
      return_code = RegOpenKeyEx ( 
         HKEY_LOCAL_MACHINE,
         "Software\\\\ODBC\\\\ODBC.INI\\\\ODBC Data Sources",
         0, 
         KEY_READ, 
         &odbc_hkey );
   }
   else {
      return_code = RegOpenKeyEx ( 
         HKEY_CURRENT_USER,
         "Software\\\\ODBC\\\\ODBC.INI\\\\ODBC Data Sources",
         0, 
         KEY_READ, 
         &odbc_hkey );
   }

if ( return_code != ERROR_SUCCESS ) {
      diagnostic_code = 2;
   }

if ( return_code == ERROR_SUCCESS ) {

// Get the first value name (dsn name) and value (driver name).

return_code = RegEnumValue(
         odbc_hkey, 
         dsn_index,
         odbc_dsn, 
         &odbc_dsn_size, 
         NULL, 
         NULL, 
         (LPBYTE) odbc_driver, 
         &odbc_driver_size );

if ( return_code == ERROR_NO_MORE_ITEMS ) {
         diagnostic_code = 3;
         RegCloseKey ( odbc_hkey );
      }
      else { 
         if ( return_code != ERROR_SUCCESS ) {
            diagnostic_code = 4;
            RegCloseKey ( odbc_hkey );
         }
      }
   }

// CHECK THE ARGUMENT NUMBERS IN THE SET_VALUE CALLS...

api_odbc_hkey.type = DT_INT;
   api_odbc_hkey.data = &odbc_hkey;
   api -> set_value ( arg_handle, 2, &api_odbc_hkey, FALSE );

api_dsn_index.type = DT_INT;
   api_dsn_index.data = &dsn_index;
   api -> set_value ( arg_handle, 3, &api_dsn_index, FALSE );

api_odbc_dsn.type      = DT_VARCHAR;
   api_odbc_dsn.data      = odbc_dsn;
   api_odbc_dsn.piece_len = ( a_sql_uint32 )( strlen ( odbc_dsn ) );
   api_odbc_dsn.len.total_len = ( a_sql_uint32 )( strlen ( odbc_dsn ) );
   api -> set_value ( arg_handle, 4, &api_odbc_dsn, 0 );

api_odbc_driver.type      = DT_VARCHAR;
   api_odbc_driver.data      = odbc_driver;
   api_odbc_driver.piece_len = ( a_sql_uint32 )( strlen ( odbc_driver ) );
   api_odbc_driver.len.total_len = ( a_sql_uint32 )( strlen ( odbc_driver ) );
   api -> set_value ( arg_handle, 5, &api_odbc_driver, 0 );

api_return_code.type = DT_INT;
   api_return_code.data = &return_code;
   api -> set_value ( arg_handle, 6, &api_return_code, FALSE );

api_diagnostic_code.type = DT_INT;
   api_diagnostic_code.data = &diagnostic_code;
   api -> set_value ( arg_handle, 7, &api_diagnostic_code, FALSE );

api_diagnostic_string.type      = DT_VARCHAR;
   api_diagnostic_string.data      = diagnostic_string;
   api_diagnostic_string.piece_len = ( a_sql_uint32 )( strlen ( diagnostic_string ) );
   api_diagnostic_string.len.total_len = ( a_sql_uint32 )( strlen ( diagnostic_string ) );
   api -> set_value ( arg_handle, 8, &api_diagnostic_string, 0 );

free ( odbc_dsn );
   free ( odbc_driver );
   free ( diagnostic_string );

} // get_first_dsn

//------------------------------------------------------------------------------------------------
// get_next_dsn
//------------------------------------------------------------------------------------------------

__declspec(dllexport) _VOID_ENTRY get_next_dsn ( an_extfn_api *api, void *arg_handle ) {

an_extfn_value  api_odbc_hkey;
   an_extfn_value  api_dsn_index;
   an_extfn_value  api_odbc_dsn;
   an_extfn_value  api_odbc_driver;
   an_extfn_value  api_return_code;
   an_extfn_value  api_diagnostic_code;
   an_extfn_value  api_diagnostic_string;

HKEY            odbc_hkey; 
   DWORD           dsn_index;

char *          odbc_dsn; 
   DWORD           odbc_dsn_size;

char *          odbc_driver; 
   DWORD           odbc_driver_size;

DWORD           return_code; 
   DWORD           diagnostic_code; 
   char *          diagnostic_string;

if( !api -> get_value ( arg_handle, 1, &api_odbc_hkey ) || api_odbc_hkey.data == NULL ) {
      diagnostic_code = 1;
      return;
   }

odbc_hkey = *(HKEY *) api_odbc_hkey.data;

if( !api -> get_value ( arg_handle, 2, &api_dsn_index ) || api_dsn_index.data == NULL ) {
      diagnostic_code = 2;
      return;
   }

dsn_index = *(DWORD *) api_dsn_index.data;

dsn_index++;

odbc_dsn = (char *) malloc( 255 );
   strcpy_s ( odbc_dsn, 255, "" );
   odbc_dsn_size = 255;

odbc_driver = (char *) malloc( 255 );
   strcpy_s ( odbc_driver, 255, "" );
   odbc_driver_size = 255;

return_code        = 0;     
   diagnostic_code    = 902;     
   diagnostic_string  = (char *) malloc( 255 );
   strcpy_s ( diagnostic_string, 255, "//-902" );

// Get the next value name (dsn name) and value (driver name).

return_code = RegEnumValue(
      odbc_hkey, 
      dsn_index,
      odbc_dsn, 
      &odbc_dsn_size, 
      NULL, 
      NULL, 
      (LPBYTE) odbc_driver, 
      &odbc_driver_size );

if ( return_code != ERROR_SUCCESS && return_code != ERROR_NO_MORE_ITEMS ) {
      diagnostic_code = 3;
      RegCloseKey ( odbc_hkey );
   }

if ( return_code == ERROR_NO_MORE_ITEMS ) {
      diagnostic_code = 4;
      RegCloseKey ( odbc_hkey );
   }

// CHECK THE ARGUMENT NUMBERS IN THE SET_VALUE CALLS...

api_dsn_index.type = DT_INT;
   api_dsn_index.data = &dsn_index;
   api -> set_value ( arg_handle, 2, &api_dsn_index, FALSE );

api_odbc_dsn.type      = DT_VARCHAR;
   api_odbc_dsn.data      = odbc_dsn;
   api_odbc_dsn.piece_len = ( a_sql_uint32 )( strlen ( odbc_dsn ) );
   api_odbc_dsn.len.total_len = ( a_sql_uint32 )( strlen ( odbc_dsn ) );
   api -> set_value ( arg_handle, 3, &api_odbc_dsn, 0 );

api_odbc_driver.type      = DT_VARCHAR;
   api_odbc_driver.data      = odbc_driver;
   api_odbc_driver.piece_len = ( a_sql_uint32 )( strlen ( odbc_driver ) );
   api_odbc_driver.len.total_len = ( a_sql_uint32 )( strlen ( odbc_driver ) );
   api -> set_value ( arg_handle, 4, &api_odbc_driver, 0 );

api_return_code.type = DT_INT;
   api_return_code.data = &return_code;
   api -> set_value ( arg_handle, 5, &api_return_code, FALSE );

api_diagnostic_code.type = DT_INT;
   api_diagnostic_code.data = &diagnostic_code;
   api -> set_value ( arg_handle, 6, &api_diagnostic_code, FALSE );

api_diagnostic_string.type      = DT_VARCHAR;
   api_diagnostic_string.data      = diagnostic_string;
   api_diagnostic_string.piece_len = ( a_sql_uint32 )( strlen ( diagnostic_string ) );
   api_diagnostic_string.len.total_len = ( a_sql_uint32 )( strlen ( diagnostic_string ) );
   api -> set_value ( arg_handle, 7, &api_diagnostic_string, 0 );

free ( odbc_dsn );
   free ( odbc_driver );
   free ( diagnostic_string );

} // get_next_dsn

[Former Member]

I'm not sure I quite grasp what you are attempting to do. It is true that if you use DT_UNSINT and DT_INT to represent an HKEY in a 64-bit binary, you can inadvertently cause bad handle errors when you pass them to windows APIs.

Are you attempting to store these HKEYs in a database? That might add even more complexity...

[Breck_Carter]

I am not storing HKEYs in a database, I am making API calls to get DSN entries from the registry. An HKEY handle is obtained by one C function and returned to the SQL code along the the first DSN ("get first DSN"), and then that HKEY is repeatedly passed to a second C function ("get next DSN) which gets the next DSN until there are no more. The error I am getting (occasionally, VERY occasionally) is "bad handle" inside the first call to the second C function. The visible result in Foxhound is that the DSN dropdown on the main menu displays only two DSNs instead of the hundreds in the registry: the first System DSN and the first User DSN, because there are two separate pieces of logic to call the first-and-second C function.

[VolkerBarth]

FWIW, a Windows HANDLE is a PVOID is a void pointer and as such a 64-bit type on 64-bit Windows, so for your questions a)-e), I would second your considerations.

AFAIK, using signed instead of unsigned types should not matter in C/C++ unless you do arithmetic/logical operations. Just assigning values with the required bitness (i.e. ones that don't have to be filled up with leading bits - where the most signifant bit will make a difference) should not lead to problems.

Nevertheless, it's recommended to chose correctly between signed/unsigned types.

[Breck_Carter]

It seems to be getting more and more reproducible :)... so perhaps I should bite the bullet and change the code and test it?

[johnsmirnios]

Yes, you are totally right. In get_next_dsn, the code will pick up 64bits of data with that line of code but there are only 32bits there. I, of course, was only ever looking at get_first_dsn. 🙂

[VolkerBarth]

All in all I'm glad you found a short answer to a somewhat long question:)

It was soooo long I put this comment by accident in the wrong position, i.e. below Martin's answer...

[MCMartin · Accepted solution]

ok, so set_value() is copying the data of api_odbc_hkey.data based on the fact that the .type is DT_INT I assume it is copying just the lower 32 Bit of the Handle (which has 64bits in your case) - John please correct me. If it is copying just the 32 bits, than the Handle value is still ok, but the cast in the second function:

odbc_hkey = *(HKEY *) api_odbc_hkey.data;

will treat api_odbc_hkey.data as being a 64bit value, so if now some garbage appears in the upper 32bits you will get your invalid Handle

[johnsmirnios · Accepted solution]

Yes, you are totally right. I had only been looking at get_first_dsn. I'm so confident, I converted your comment to an answer ... we can revert it if we are wrong.

[johnsmirnios · Accepted solution]

I would add that it would be preferable to use the actual native types everywhere: do not truncate handles to 32-bit and do not rely on a specific endian.

[Breck_Carter · Accepted solution]

I thought I had a reproducible test program, but it is unreliable; i.e., it ran overnight without failing 🙂

Until I can reproduce the symptom at will, it's pretty hard to prove that a fix worked... I'm sure you've heard that before.

[Breck_Carter · Accepted solution]

...still no reliable repro

[Breck_Carter · Accepted solution]

...ok, now I have a reliable repro, work may proceed.

[MCMartin]

A good article is this: Pushing the Limits of Windows: Handles It shows, that even as the handle has a size of 64bit (in 64bit process) it is just the index in the process handle table, which itself is limited to nearly 16Mio entries.

So you can transfer a handle from a 64bit to a 32bit process and as the follwing article explains it is more or less resized to fit the bitsize of the target process: Transfering a pointer across processes

So for me it would be formally correct to use the bigint in a 64bit runtime for the handle, but you should be still safe if you just ignore this (because the value will never be >16Mio). You could even do it vice versa and use always the bigint type to store handles (even in a 32 bit runtime).

[Breck_Carter]

Is it possible that the REALLY FUNKY mechanism for passing parameters in calls from SQL to external C procedures has something to do with it?

Plus the fact that the 64-bit C function is running in the same address space as the 64-bit SQL Anywhere server... it is not passing stuff between 32-bit and 64-bit processes.

[johnsmirnios]

Generally, yes, HANDLEs fit within 32bits and the SA shared memory port actually relies on that fact so that 32-bit and 64-bit processes can communicate over the same port. However, I would also be cautious about where the handle came from. Straight from the OS as you have above, the values will be small but if, for example, you obtained a connection "handle" from ODBC it may not have the same properties. The ODBC driver manager does handle-mapping so that it can figure out from a handle which driver you are talking to. It would theoretically be free to use the high bits.

[Breck_Carter]

Could it be the fact the C routine is treating the 32 bit input value as 64 bits and picking up 32 bits of garbage (usually zero but sometimes not?)

Remember, this is the funky external procedure call parameter-passing mechanism we're talking about... the mind-numbing get_value and set_value functions described in the SQL Anywhere Help.

[johnsmirnios]

On a little-endian machine, the first 4 bytes are the low order bytes so, no, you shouldn't be picking up junk values. It's possible that registry handles don't conform to the other OS handle rules. You could just put a check in your code to verify that HIDWORD((DWORD_PTR)odbc_key) == 0. If the high bits are ever set, this check will fail. You could also verify your theory by just checking the high dword in the debugger without changing the code.

[Breck_Carter]

I have separate SQL code for calling separate 32-bit versus 64-bit dlls, so each version is standalone and under my complete control. The SQL code decides which flavor of SQL Anywhere is running, and does a different CREATE PROCEDURE at runtime depending on the answer. The 32-bit version works, I just need to fix the 64-bit path through the code.

[johnsmirnios]

If the high bits are set, you will definitely need to change the code. I'm just suggesting you could confirm whether the change is necessary by watching for some high bits being set.

[MCMartin]

With the full story I think I understand your problem, it is the usage of odbc_hkey in your first function, it is a local variable allocated on the stack.

You provide the pointer (not the data) into your outgoing parameter:

api_odbc_hkey.data = &odbc_hkey;

This means, that after the call to this function you are using in the second call a pointer to an already freed memory. The freed memory occassionally is overwritten or not leading to the observation that you have, that is sometimes reports an invalid handle (in the overwritten case)

What you would have to do is use a static variable (if your are sure that only one thread can access the two methods) or

better solution is something like this:

api_odbc_hkey.data = new HANDLE; (or use malloc or whatever...)
memcpy(&odbc_hkey,api_odbc_hkey.data,sizeof(HANDLE));

and after get_next_dsn doesn't find anymore DSNs you will have to free that memory !

[Breck_Carter]

FWIW the 32-bit version of this code has functioned properly for many many years, so if it a problem with freed memory then my luck has been extraordinary.

The code required to return a value isn't just one statement, it is three:

api_odbc_hkey.type = DT_INT;

api_odbc_hkey.data = &odbc_hkey;

api -> set_value ( arg_handle, 2, &api_odbc_hkey, FALSE );

This is the technique shown in the SQL Anywhere Help; do a search on set_value to see.

[Breck_Carter]

To continue... api_odbc_hkey.data is expecting a pointer...

typedef struct an_extfn_value {

void * data;

See http://dcx.sybase.com/index.html#1201/en/dbprogramming/pg-extfun-value.html

With some data types you have to use &, with others not. Here is an example from the sample extproc.cpp file shipped with SQL Anywhere 12...

char        result[256];
...
retval.type = DT_FIXCHAR;
retval.data = &result;
retval.piece_len = retval.len.total_len = strlen( result );
api->set_value( arg_handle, 0, &retval, 0 );

[johnsmirnios]

The api->set_value() call will copy the value out. We don't just save the pointer that you provided.

[VolkerBarth]

Agreed, that's how my (32-bit) DLLs use set_value with local variables, too, as does the ExternalProcedures sample.

Therefore I guess the funky external interface has to copy the data by itself.

[VolkerBarth]

...and John has just verified my guess:)

[Breck_Carter]

Yabbut... when it is received in the second C routine, what happens then? That is where the "bad handle" error occurs.