Girish Jain on .Net Framework

I will be writing about my favorite technology which is Microsoft .Net Framework and how to use it to improve developer productivity

CLR Journey from PInvoke to WinRT

clock June 21, 2013 11:20 by author Girish Jain

 

In this blog post, I am going to talk about Microsoft .Net Common Language Runtime (CLR) journey from its early days to latest Windows Runtime (WinRT) platform, which is released with Windows 8. I would suggest you read my previous blog post on WinMD files before reading further to gain better understanding of WinMD files introduced with Windows 8.

WinRT is the new platform released with Microsoft’s latest operating system Windows 8 and .Net Framework 4.5. It is important to remember that for WinRT APIs to work it needs at least Windows 8 as operating system (or Windows Server 2012 for server family of operating systems), and hence, WinRT is not available for previous platforms.

Coming back to WinRT, did we need another set of runtime libraries and a new file extension, WinMD? I am sure you are having this basic question in mind for what good this new WinRT library is going to do and why on earth Microsoft added a new file type? I will try and answer it in this blog post.

 

Windows Runtime (WinMD at heart)

WinRT is the new platform which exposes underlying operating system APIs in object-oriented manner to developers (across languages) and enables them to make best use of underlying platform capabilities to build great immersive apps.

WinMD files are at the heart of new Windows Runtime (WinRT) concept. These new WinMD files only contain metadata for WinRT types. WinMD files themselves do not contain any code. Hence, when you open any WinMD file using ILDAsm tool you see that all class methods are marked as "runtime managed" which, as per ECMA-335 CLI specification, means the managed implementation for the method will be provided by CLR. Please read my earlier blog post for more details on WinMD files.

Underneath, CLR bridges the gap of converting managed calls to low-level operating system (native) API calls and vice versa i.e. invoking managed callback functions from native world. This is a great step forward now because CLR is taking care of lot of complexities related to making these native API calls from managed code and vice-versa. Native APIs are written as C/C++ libraries, and before WinRT was introduced, these were consumed using Platform Invoke feature of CLR, commonly referred to as PInvoke.

With the introduction of WinRT, there has been a paradigm shift for managed developers for how they consume platform capabilities which otherwise are not already wrapped up in BCL. Let’s understand this evolution starting from its early days to latest WinRT platform. This will help you understand the internal workings of CLR and role of WinMD files too. We have come a long way with introduction of WinRT, it’s time for a little flashback.

 

.Net is Born!

To be able to best understand why we need Windows Runtime (WinRT) and WinMD files or how it is helpful, we need to go back a little into history and understand what kind of problems were there, before WinRT was born, which are now addressed by it. With the introduction of CLR, Microsoft developed rich metadata to describe types and their members in libraries (which is defined in ECMA-335 spec). This was a great step forward as existing libraries (DLLs) did not have a way to manifest it in themselves but we had to separately produce type library (TLB) files for components so that these components could be consumed in other applications. For example, you could write a COM component in C++ and then consume it in VB6 application. With the advent of .NET, assemblies themselves contained code as well as rich metadata, which was great as assemblies could describe its own types, members, and the external assemblies that it references (this was key feature which solved lot of DLL referencing related issues along with CLR assembly binder), which made assemblies self-contained and deployable unit. Hence, referencing and consuming these assemblies from other applications was a lot easier as compared to olden days, where referencing correct version of DLL, building code, and deploying binaries itself was a big challenge, commonly known as “DLL Hell”. The single biggest factor which helped solve DLL hell problem in .Net world was the ability to manifest all the rich metadata within the assembly (library) itself. Traditionally, components had very tight binary linking whereas CLR assemblies had metadata based linking.

 

PInvoke is born (Re-use existing investment)

To begin with, .Net developers did not have everything in the form of these assemblies, especially when it came to underlying low-level operating system APIs (such as APIs exposed by kernel32.dll) because, these were written and available in the form of C/C++ libraries. Microsoft did wrap certain APIs with managed wrappers over native Windows APIs, such as those for logging entries to event viewer, EventLog*classes in System.Diagnostics namespace which, are just managed wrappers over underlying native APIs. But, Microsoft could not have done it for all APIs (simply because there would be no value addition with creating and maintaining tons of managed wrappers over native APIs).

Therefore, Microsoft provided a way to enable .Net developers to consume these libraries using PInvoke. The way PInvoke works is that you would define API method signature in code and mark it as external i.e. the method implementation being available in another DLL (mentioned using attributes). Underneath, CLR would create Runtime Callable Wrapper (RCW) stub which would take care of marshalling calls between .Net Apps and native libraries. CLR would rely on method signature which you re-create in code to take care of marshalling requirements. Methods, marked as external, and type definitions re-created in source code helped CLR get the required metadata for it to be able to generate the RCW stub for marshalling and forwarding calls to libraries. In essence, metadata for native types/structures and methods was key input for CLR to make its magic and connect the two worlds.

 

PInvoke Limitations

PInvoke was great but it had its own limitations because there was still considerable amount of work to be done for .Net developers to recreate types in code (which were needed for the API call), define marshalling rules for these types, their memory layout, and so on (which was ugly and to make even a basic API call work, it took considerable amount of effort). Lots of these underlying APIs work with pointers whereas .Net developers don't directly work with pointers so they struggled with making API calls using PInvoke (working with IntPtr did not come easy to VB.Net or C# developers, to say the least). Although, they used familiar programming language such as VB .Net or C# but concepts were alien, such as pointers, memory layout, etc.

With PInvoke, CLR would use types and method signature created in source code to derive metadata, which it needed to create RCW stub, required for marshalling the call to native library. Lot of problems with this approach were derived from the fact that developers found it difficult to re-create method signatures and types in C#/VB code equivalents as they were required by CLR. In fact, there was a site (http://pinvoke.net) available, just for managed developers, to help with signature of these native methods and types such that these can be specified in their C#/VB counterparts, and this site proved to be a great asset for .Net developers trying to make native API calls using PInvoke. I am sure every developer who has created CLR applications, would have used this site at some point in time.

Therefore, Microsoft provided a way to enable .Net developers to consume these libraries using PInvoke. The way PInvoke works is that you would define API method signature in code and mark it as external i.e. the method implementation being available in another DLL (mentioned using attributes). Underneath, CLR would create Runtime Callable Wrapper (RCW) stub which would take care of marshalling calls between .Net Apps and native libraries. CLR would rely on method signature which you re-create in code to take care of marshalling requirements. Methods, marked as external, and type definitions re-created in source code helped CLR get the required metadata for it to be able to generate the RCW stub for marshalling and forwarding calls to libraries. In essence, metadata for native types/structures and methods was key input for CLR to make its magic and connect the two worlds.

For example:

[DllImport("kernel32.dll", CharSet=CharSet.Auto, SetLastError=true)]
internal static extern IntPtr GetProcAddress([In] IntPtr hModule, [In, MarshalAs(UnmanagedType.LPStr)] string lpProcName);

 

WinRT to Rescue (Partially)

.Net developers had to re-create metadata in source code such as method signatures and type definitions for CLR to do RCW magic. And this was the biggest pain area for most developers. This is where WinRT makes life easy for developers as Microsoft came up with these new WinMD files which define WinRT libraries classes and methods in rich metadata format supported and understood by CLR . Now, developers don’t need to re-create method signatures in code but just add WinMD files as reference to the project.

Now, with following three things falling in place with Windows 8 (and Windows Server 2012),

1. underlying operating system providing runtime libraries (WinRT), exposing native APIs in object-oriented manner

2. rich metadata available for WinRT classes from WINMD files and

3. RCW magic of CLR already in place,

Microsoft added new WinRT Interop functionality (on top of COM Interop infrastructure) to CLR and made life easy for managed developers. Now we can consume WinRT APIs just as you would consume any other managed API (in natural and familiar manner). With Windows Runtime you don’t have to deal with pointers and lot of complexities related with native Windows APIs but they are dealt for you by CLR. You still write code in managed language which you are familiar with.

So, WinMD files enable developers to consume native APIs from .Net apps in a natural and familiar manner (which they otherwise had to do painfully using PInvoke). So this solved an existing pain area for developers and then Microsoft went a step further which enabled developers to consume these native APIs from an array of programming languages, which are traditionally not used for .Net apps development, such as JavaScript by introducing a new feature called as "language projection".

So, to answer the question as WHY Microsoft introduced new WINMD files - it was to manifest rich metadata for operating system APIs so that developers can consume these platform APIs from managed code in a natural and familiar manner. This is the reason why Microsoft says that WinRT is NOT a new layer in itself BUT only exposes underlying operating system APIs to managed world developers.

 

Does WinRT replace PInvoke?

WinRT does not replace PInvoke as WinRT does not cover every API supported by native Windows libraries and hence, for certain APIs you would still need to use PInvoke. Rather, WinRT is enhanced version of PInvoke minus its complexities. WinRT builds on top of existing PInvoke and COM Interop infrastructure and build on existing concepts of RCW and CCW, which is great as it is re-using existing infrastructure and at the same time reducing complexities of it.

 

WinRT only for .Net?

Windows Runtime libraries (for e.g. Windows.UI.dll) are not implemented in managed code but some low-level language (my guess is C++) and made available to managed code using Windows Runtime Interop (built on top of COM Interop infrastructure) and .winmd files. The types which we see in WinMD files are defined and actually implemented by these WinRT libraries. But is WinRT only available for .Net developers? Not really, WinRT libraries can be accessed from outside CLR as well, for example, when you use JavaScript to build Metro apps, you are not using CLR to host your application but “Chakra” engine, which is used by Internet Explorer too, and internally it is consuming these WinRT libraries. So that proves the point that WinRT is not just for managed developers.

In fact, you can create classic-style COM components using Windows Runtime C++ Template Library (WRL) which can consume WinRT APIs and the COM component can then be accessed from any COM enabled technology including classic desktop apps.

 

WinRT Duplicates APIs?

Why Windows Runtime libraries duplicate certain sets of classes, especially those related to UI controls and thread pool when these are already available with .Net Framework? Why this duplication? – This duplication is there for a reason. As we saw above, WinRT is not just for managed developers but is consumed from other application hosts as well and it became necessary to make these basic sets of classes available through WinRT too. Managed developers had some of these classes already available to them and it looks like duplication for them but these classes exist in WinRT for a reason.

 

WinRT for Desktop Apps

Only a subset of WinRT Classes/APIs are available to desktop apps. Also, it is important to remember that WinRT is only available Windows 8 onwards so your target platform for the desktop app must be at least Windows 8 for the ability to access WinRT APIs.

S Hanslman has written this excellent blog post which shows manual steps for how to access WinRT APIs from desktop apps.

 

Summary

Microsoft has introduced new set of runtime libraries for Windows platform, commonly referred to as WinRT library, which expose native functionality in object oriented manner. These new libraries are based on solid principles of COM but Microsoft has shielded developers from complexities of COM. To make it easy for developers to interact with this underlying platform library, Microsoft developed WinMD files and supporting interop functionality to different runtime environments such as CLR and Chakra engine so that developers can consume these libraries from different languages (Java Script, VB .NET, C#, C++, etc) in natural and familiar manner. WinRT and its interop with runtime environments makes platform capabilities available to an array of programming languages in natural and familiar manner.

 

In my next blog post, I will try to go further into this new magical world of WinRT and further explore its internal working.

Happy Coding.

Vande Mataram!

(A salute to motherland)

P.S. In addition to blogging, I use Twitter to share tips, links, etc. My Twitter handle is: @girishjjain




WINMD Files Under the Hood

clock October 13, 2012 00:04 by author Girish Jain

WINMD Files

I have recently installed Windows 8 on my laptop and am having a go with new Windows 8 operating system and the newly introduced Windows Runtime (WinRT) layer. As part of this blog post, I am going to dig deep into WINMD files, which are at the heart of this new WinRT layer.

 

WINMD File Format

First of all I wanted to understand more about file format of WINMD files. I wanted to know more internal detail about this new file extension so I decided to check it with the old friend DUMPBIN.exe utility using following command:

D:\DDrive\Girish\temp>dumpbin /HEADERS /CLRHEADER 
"C:\Windows\System32\WinMetadata\Windows.System.winmd" /OUT:output.txt

 

Great news, it worked with dumpbin.exe! I was happy to see that it worked with dumpbin.exe tool (as that confirmed that the file format is familiar) and the tool did give lot of information about the file as well. Find below the output of same, I have highlighted certain key aspects of same:

Dump of file C:\Windows\System32\WinMetadata\Windows.System.winmd

PE signature found

File Type: DLL

FILE HEADER VALUES
             14C machine (x86)
               1 number of sections
        5010588C time date stamp Wed Jul 25 21:35:24 2012
               0 file pointer to symbol table
               0 number of symbols
              E0 size of optional header
            2102 characteristics
                   Executable
                   32 bit word machine
                   DLL

OPTIONAL HEADER VALUES
             10B magic # (PE32)
           11.00 linker version
               0 size of code
               0 size of initialized data
               0 size of uninitialized data
               0 entry point
               0 base of code
               0 base of data
          400000 image base (00400000 to 00405FFF)
            1000 section alignment
             200 file alignment
            6.02 operating system version
            0.00 image version
            6.02 subsystem version
               0 Win32 version
            6000 size of image
             200 size of headers
           127B8 checksum
               3 subsystem (Windows CUI)
             540 DLL characteristics
                   Dynamic base
                   NX compatible
                   No structured exception handler
          100000 size of stack reserve
            1000 size of stack commit
          100000 size of heap reserve
            1000 size of heap commit
               0 loader flags
              10 number of directories
               0 [       0] RVA [size] of Export Directory
               0 [       0] RVA [size] of Import Directory
               0 [       0] RVA [size] of Resource Directory
               0 [       0] RVA [size] of Exception Directory
               0 [       0] RVA [size] of Certificates Directory
               0 [       0] RVA [size] of Base Relocation Directory
               0 [       0] RVA [size] of Debug Directory
               0 [       0] RVA [size] of Architecture Directory
               0 [       0] RVA [size] of Global Pointer Directory
               0 [       0] RVA [size] of Thread Storage Directory
               0 [       0] RVA [size] of Load Configuration Directory
               0 [       0] RVA [size] of Bound Import Directory
               0 [       0] RVA [size] of Import Address Table Directory
               0 [       0] RVA [size] of Delay Import Directory
            1000 [      48] RVA [size] of COM Descriptor Directory
               0 [       0] RVA [size] of Reserved Directory


SECTION HEADER #1
   .text name
    45D0 virtual size
    1000 virtual address (00401000 to 004055CF)
    4600 size of raw data
     200 file pointer to raw data (00000200 to 000047FF)
       0 file pointer to relocation table
       0 file pointer to line numbers
       0 number of relocations
       0 number of line numbers
40000020 flags
         Code
         Read Only

  clr Header:

              48 cb
            2.05 runtime version
            1050 [    4580] RVA [size] of MetaData Directory
               1 flags
                   IL Only
               0 entry point token
               0 [       0] RVA [size] of Resources Directory
               0 [       0] RVA [size] of StrongNameSignature Directory
               0 [       0] RVA [size] of CodeManagerTable Directory
               0 [       0] RVA [size] of VTableFixups Directory
               0 [       0] RVA [size] of ExportAddressTableJumps Directory
               0 [       0] RVA [size] of ManagedNativeHeader Directory


  Summary

        5000 .text

 

As you can see from the highlighed parts, it is clear that:

1. WINMD files are standard PE/COFF format files.

2. WINMD files are standard Win-32 DLL files.

3. WINMD files contain CLR headers as well which means WINMD files comply with ECMA-335 CLI specification for file format and hence, contain rich metadata (which means they are not just native Win 32 DLLs but contain rich metadata just like any other CLR assembly)

In simple terms, WINMD files are DLL files (or CLR assemblies) with rich .Net framework metadata to expose underlying Windows operating system APIs.

 

Disassemble WINMD Files

Now we know that WINMD files comply ECMA-335 CLI specification for file format and hence, they are .Net assemblies. In that case, we should be able to open these using ILDAsm (IL Disassembler) tool or Reflector. I tried to open Windows.System.winmd file and I was able to open it in ILDASM, refer below screenshot:

Remember, you have to use latest version of ILDAsm supplied with Visual Studio 2012 as previous versions would not work with WINMD files.

Excellent! Now you would think that you can browse through all the types and methods and method IL code but there is a surprise. When you try to open any method body you will get to see below:

.method public hidebysig newslot virtual final 
        instance void  Enable() runtime managed
{
  .override Windows.System.Threading.Core.ISignalNotifier::Enable
} // end of method SignalNotifier::Enable

 

You will notice that there is no IL code in the method body and the method is marked with runtime managed flag, to understand the meaning of the flag we will need to refer ECMA-335 Common Language Infrastructure (CLI) specification.

Find below extract from ECMA-335 Common Language Infrastructure (CLI) specification. for method flags:

FlagsValueDescription
Runtime 0x0003 Method impl is provided by the runtime
Managed 0x0000 Method impl is managed
... ... ...

 

It tells us that runtime managed means method does not have a body AND the managed implementation will be provided by runtime. This brings us to following conclusion that WINMD files (under C:\Windows\System32\WinMetadata folder) do not contain code but metadata only. Since these files do not contain code but rich metadata only hence, that also explains why these files are so small in size.

 

WINMD and no code?

Does it mean that WINMD files cannot contain code? Not really. Usually, WINMD files will not contain code but not that they cannot. Usually, WINMD files will only contain rich metadata but if you create a new custom windows runtime component of your own using Visual Studio 2012, it will produce a WINMD file and it will contain both, code and rich metada and you will be able to verify the same using ILDAsm tool.

 

Conclusion

1. WINMD stands for Windows Meta Data and as the name suggests these files contain rich metadata for Windows operating system APIs, making these APIs available to managed world.

2. WINMD files use DLL file format but with a different extension. Question is why did Microsoft choose to create a new file extension (WINMD) and not reuse existing DLL extension? I believe the reason would be that WINMD files can contain just metadata unlike traditional DLL files which are generally used as library of functions and therefore contain code too. Nonetheless, WINMD files are DLL files.

3. WINMD files contain CLR header and rich metadata about its types, methods, etc. and hence, these files are .Net Framework assemblies. This makes Windows APIs available to managed world in a natural and familiar manner.

4. Lastly, as we saw that these meta data only files do not contain code, it tells me that WinRT is NOT a new layer in itself but, exposes the underlying Windows operating system API layer to managed world (more about this in future post).

In my next blog post, we will go further into this new world of Windows Runtime (WinRT) introduced with Windows 8.

Happy Coding!!

Vande Mataram!

(A salute to motherland)

P.S. In addition to blogging, I use Twitter to share tips, links, etc. My Twitter handle is: @girishjjain




About the author

Girish Jain works on Microsoft .Net framework technologies and is a big fan of WPF, WCF, and LINQ technologies. He is currently based in India with his wife and a daughter. When not spending time with family, Girish enjoys creating small tools, utilities, frameworks to improve developer productivity.

Sign In