Pierre Schweitzer’s Blog

Indeed, now 2010 is over, it may be interesting to go back to it. Not for the same reasons that led the ReactOS to go back to 2010 (cf: r50529, a revert needed due to a bug on boot image handling). In fact, it is interesting to go back to 2010, just to see what has been done, what was planned, what has failed. Finally, what we can say and remember about that year.

First of all, I would like to speak about my servers. As you may know (or not) I currently own two servers, known as www.heisspiter.net and www2.heisspiter.net, and I also rent a third server at OVH, called www3.heisspiter.net. I was not speaking about them, because I was not really taking care of them. And this led to many and important issues (bad performances, bots, and so on). During a weekend, I decided to switch www3 to ipv6, which finally worked. But, it made me understand that my servers needed love, somehow. So, now, I am working back on them. The idea that they have reached some maturity point and can work alone is definitely wrong.  Indeed, 2009 was already a pretty bad year for heisspiter.net and 2010 was terrible. No evolution, several issues, servers down, … A quick look at statistics show that people also stop coming on the server. And I cannot blame them. 2010 was a really bad year for heisspiter.net, and 2011 cannot be worse. I will just do my best not to make it the same.
An encouraging note regarding heisspiter.net, at least. Some evolutions have started. I was talking about ipv6, but also a webmail for users arrived, upload service is back, mail server has been fixed, servers software have been updated, and heisspiter.net internal tools fixed. This actually explains why heisspiter.net is more stable since December!

Other major point… Of course, ReactOS! You may have seen, reading my other posts that the project did important step into stability and features. Some rewrites, some parts becoming more mature. MM rewrite, with Heap rewrite (for user-mode land) force developers to fix the ReactOS code to corrupt less memory (or to less corrupt memory?). And this works. Especially when fixes are applied to those rewrites. This also comes after some hard year for ReactOS, with no releases, and nothing to release, due to broken trunk. But, here, it is past!
My modest goal, for 2011, is to prove that ReactOS has gained some maturity now. Some testers are already pushing to get 0.4, and I would like to show we are not that far. And I will try to show it on my domain of work on ReactOS, ie filesystems and kernel. With the help of Johannes Anderwald, and Art Yerkes, I attempt to make ReactOS boot from Microsoft FastFAT driver. Johannes has brought some code for tunnels handling in FsRtl, Art code for MCBs and CC. Finally, I come with notifications (still) and motivation to make all that stuff working together (which is not, at the moment). This is a very, very interesting experiment since it kinda stresses ReactOS and forces me to work on ReactOS part I promised I would never work on (I am speaking of CC!). On another side, I will also keep on working on other parts of the kernel as I did previously, trying to improve it and match Windows 2003. I will also switch a bit on FreeLDR, some bugs are calling me there!

About personal projects, I have been quite active during 2010, even if I did not publish about them. One of the project I wanted to publish about before I forgot is a C++ garbage collector. I designed it for several uses, and finally it is more a memory manager than a garbage collector. Its purpose is simple: giving you memory whenever you need it and keeping track about it. It can also performs some operations on it to make your program debug easier such as: memory marking, memory zone tagging. It can also allocate non-paged memory, check against corruption, and so on. It has been designed to work in multi-threaded environment and provides functions for that. For example, when you share memory zones between threads, sometimes you even do not recall who is using what, how long. Here garbage collector becomes useful. Each threads when it uses a zone just needs to reference the memory zone. And once it is done, it dereferences it. Simple mechanism, but that ensure the memory zone will be released once every thread is done with it.
I am not totally done with that project (that is perhaps why I did not publish about it yet) and I plan to finish it and make it a bit closer to a garbage collector. And giving it the ability to allocate and release objects.

Other project I have been working on (and I am still actively working on) is an IRCd “new generation” written in C++. Its purpose is quite simple, implementing the five RFC concerning IRC, optionally adding extra often used/needed features. But, the new thing is that it comes with services implemented in (if built with, of course!). This is quite new, and interesting in my opinion. When you need to rapidly deploy an IRCd, configuring both IRCd, services (when you found the good ones!) can be a pain. With that IRCd, everything comes in. Thus it makes services really efficient as they directly communicate with the server (in a proper way, nothing messy!). And there is no need for SVSMODE, SVSJOIN, etc, commands or equivalents, here you just use services. At the moment, the core IRCd is almost complete and works really well. Services are mostly non existant (excepted OpServ, obviously).
For 2011, I plan to finish that IRCd, and perhaps to use it on heisspiter.net. Time will tell.

Finally, this is the shortened version, but there would be so much to say… Best thing is to keep reading ReactOS’ mailing-lists and this blog to keep informed!

Happy new year ;).

Let’s first define what a PDO is. PDO is an acronym for Physical Device Object. A PDO is a kernel object to describe a real physical device (contrary of logical or virtual device). It’s represented in kernel space by a structure called DEVICE_OBJECT (because it applies to all device objects). And then this structure is shared between kernel and drivers. Here is the structure
typedef struct _DEVICE_OBJECT {
  CSHORT Type;
  USHORT Size;
  LONG ReferenceCount;
  PDRIVER_OBJECT DriverObject;
  struct _DEVICE_OBJECT* NextDevice;
  struct _DEVICE_OBJECT* AttachedDevice;
  PIRP CurrentIrp;
  PIO_TIMER Timer;
  ULONG Flags;
  ULONG Characteristics;
  volatile PVPB Vpb;
  PVOID DeviceExtension;
  DEVICE_TYPE DeviceType;
  CCHAR StackSize;
  union {
    LIST_ENTRY ListEntry;
    WAIT_CONTEXT_BLOCK Wcb;
  } Queue;
  ULONG AlignmentRequirement;
  KDEVICE_QUEUE DeviceQueue;
  KDPC Dpc;
  ULONG ActiveThreadCount;
  PSECURITY_DESCRIPTOR SecurityDescriptor;
  KEVENT DeviceLock;
  USHORT SectorSize;
  USHORT Spare1;
  PDEVOBJ_EXTENSION DeviceObjectExtension;
  PVOID Reserved;
} DEVICE_OBJECT, *PDEVICE_OBJECT;

But before using it, kernel must check whether it received a good PDO. Why kernel? Because most generally PDO is allocated by a driver using IoCreateDevice, which fails in case of error and driver can stop properly. If it doesn’t fail, driver keeps pointer in memory and doesn’t need to check it further. But kernel is receiving such pointers from drivers and must ensure they are good (for example, checking they are not random memory addresses). So, what does kernel do to check those pointers? First, it checks for a non-null pointer. No need to continue if pointer is null. It doesn’t check whether PDO pointer is valid, it checks one of it’s member. As you can see upper, there’s a pointer in DEVICE_OBJECT to a DEVOBJ_EXTENSION structure. Internally, an other structure is used: EXTENDED_DEVOBJ_EXTENSION. It extents the “normal” structure, giving more informations, used by the kernel.
This structure looks like that
typedef struct _EXTENDED_DEVOBJ_EXTENSION
{
  // …
  ULONG ExtensionFlags;
  PVOID DeviceNode;
  struct _DEVICE_OBJECT* AttachedTo;
  // …
} EXTENDED_DEVOBJ_EXTENSION, *PEXTENDED_DEVOBJ_EXTENSION;
This structure provides an interesting opaque pointer to a DEVICE_NODE structure. This structure, that driver should never change, is the one used by kernel to check the validity of the PDO. So, it first checks if the DeviceNode pointer is null. If not, it checks the Flags member of the structure to find if the flag DNF_ENUMERATED is set (which ensure that the PDO has been correctly initialised). When those two conditions are verified, the PDO is considered as valid.

In case of a failed test, what does kernel do? In fact, there a two different way to handle that. A cool one, for none important cases, and a more drastic one for cases where PDO must be valid. When a non valid PDO is received in a not “important” function, it can just leave with a STATUS_INVALID_DEVICE_REQUEST status and let called handle this case. In important cases, kernel must stop Windows execution to prevent any problems. Only one solution, the call to KeBugCheckEx function (producing BSOD…). KeBugCheckEx is called with specific parameters to help debugging. First parameter, the BugCheck code is set to 0xCA. It’s to indicate that it’s the PNP manager (part of the IO branch) that encounters an error and that it can’t recover it. Then come the first BSOD parameter, 0×2 it’s to indicate that PNP manager received an invalid PDO. And finally, we put the PDO address (to be able to have more informations using WinDbg). Then MSDN speaks about a third parameter. The experience shows that Windows (at least XP) doesn’t fill it.

Here is the way we could see that in C. First, we’ll define a helper macro to check the PDO:
#define IopIsValidPhysicalDeviceObject(PhysicalDeviceObject)
((((PEXTENDED_DEVOBJ_EXTENSION)PhysicalDeviceObject->DeviceObjectExtension)->DeviceNode) && (((PEXTENDED_DEVOBJ_EXTENSION)PhysicalDeviceObject->DeviceObjectExtension)->DeviceNode->Flags & DNF_ENUMERATED))
In this macro, we check the PDO as Windows kernel does. Why Iop prefix? Io because it’s part of the IO branch, and p because it’s a private function. And then let’s see the two cases, first one with return:
NTSTATUS IoSomeFunction(PDEVICE_OBJECT PhysicalDeviceObject)
{
  if (!IopIsValidPDO(PhysicalDeviceObject))
  {
    return STATUS_INVALID_DEVICE_REQUEST;
  }
  // …
}
And the second case:
VOID IoSomeFunction(PDEVICE_OBJECT PhysicalDeviceObject)
{
  if (!IopIsValidPDO(PhysicalDeviceObject))
  {
    KeBugCheckEx(PNP_DETECTED_FATAL_ERROR, 0×2, PhysicalDeviceObject, 0, 0);
  }
  // …
}

FsRtl, understand Filesystem Runtime library, is a part of ntoskrnl (Windows and ReactOS kernel). It provides a set of advanced functions for FSD, FileSystem Drivers, to interact easily with system. For example, most used function by FSD writers must the FsRtlNotify* functions. Those ones, called by the driver, notify kernel (and then user) that a change occurs on volume. It can be change on files (added, deleted, modified, etc.) or a change on volume status (locked, dismounted, etc.). As you can see, those functions aren’t “essentials”; a FSD can be written without them, it will just lack some features.

That’s why they weren’t a priority in ReactOS development, so most of them aren’t implemented, and producing BSOD, Blue Screen Of Death. Indeed, in kernel, in order to preserve system integrity, when a call to an unimplemented functions is done, it produces a BSOD. For example, FsRtlNotifyInitializeSync is written that way in kernel:

VOID
NTAPI
FsRtlNotifyInitializeSync(IN PNOTIFY_SYNC *NotifySync)
{
    KEBUGCHECK(0);
}

It’s KEBUGCHECK macro that produces the BSOD (taking is name from KeBugCheck function). But why have been those functions ignored? First of all, because of their non-vital state, but also because of their lack of documentation. When a function is documented, and all its components released it’s easier, reading MSDN to find an implementation. But FsRtl was really badly documented (and even if efforts have been done, it’s still not enough). Moreover, FsRtl functions are often using parameters that Microsoft calls “opaque” pointers. Concrete, those are pointers to internal structures. Kernel deals with them and FSD are just keeping their pointers to call functions later. So, we point to unknown elements. When writing FSD that’s not a problem, and even more that can be an advantage: kernel do everything. But, for implementation, it becomes a real problem because those internal structures (such as NOTIFY_SYNC) aren’t documented. Some of the undocumented structures can be found with WinDbg, but FsRtl ones can’t be. Only remains one solution: reverse engineering. That way, we can find how ntoskrnl work with those structures, and we can try finding what for are the field designed.

For now, on my FsRtl work, no internal structure has been added. I’ve to become a bit better . Anyway, using the above described methods, I managed to implement, in my branch pierre-fsd, many FsRtl functions, this makes kernel a bit more closer to Windows one. Indeed, I’m pretty proud to say that all Dbcs and Name functions have been implemented and working! But new problems are coming… Last function I implemented is FsRtlNotifyVolumeEvent. And it calls (as Windows one) a Io, Input/Output, function from kernel: IoReportTargetDeviceChangeAsynchronous. And this functions isn’t implemented! BSOD has just been moved…

So adventure has to be continued