Pierre Schweitzer’s Blog

Indeed, now 2010 is over, it may be interesting to go back to it. Not for the same reasons that led the ReactOS to go back to 2010 (cf: r50529, a revert needed due to a bug on boot image handling). In fact, it is interesting to go back to 2010, just to see what has been done, what was planned, what has failed. Finally, what we can say and remember about that year.

First of all, I would like to speak about my servers. As you may know (or not) I currently own two servers, known as www.heisspiter.net and www2.heisspiter.net, and I also rent a third server at OVH, called www3.heisspiter.net. I was not speaking about them, because I was not really taking care of them. And this led to many and important issues (bad performances, bots, and so on). During a weekend, I decided to switch www3 to ipv6, which finally worked. But, it made me understand that my servers needed love, somehow. So, now, I am working back on them. The idea that they have reached some maturity point and can work alone is definitely wrong.  Indeed, 2009 was already a pretty bad year for heisspiter.net and 2010 was terrible. No evolution, several issues, servers down, … A quick look at statistics show that people also stop coming on the server. And I cannot blame them. 2010 was a really bad year for heisspiter.net, and 2011 cannot be worse. I will just do my best not to make it the same.
An encouraging note regarding heisspiter.net, at least. Some evolutions have started. I was talking about ipv6, but also a webmail for users arrived, upload service is back, mail server has been fixed, servers software have been updated, and heisspiter.net internal tools fixed. This actually explains why heisspiter.net is more stable since December!

Other major point… Of course, ReactOS! You may have seen, reading my other posts that the project did important step into stability and features. Some rewrites, some parts becoming more mature. MM rewrite, with Heap rewrite (for user-mode land) force developers to fix the ReactOS code to corrupt less memory (or to less corrupt memory?). And this works. Especially when fixes are applied to those rewrites. This also comes after some hard year for ReactOS, with no releases, and nothing to release, due to broken trunk. But, here, it is past!
My modest goal, for 2011, is to prove that ReactOS has gained some maturity now. Some testers are already pushing to get 0.4, and I would like to show we are not that far. And I will try to show it on my domain of work on ReactOS, ie filesystems and kernel. With the help of Johannes Anderwald, and Art Yerkes, I attempt to make ReactOS boot from Microsoft FastFAT driver. Johannes has brought some code for tunnels handling in FsRtl, Art code for MCBs and CC. Finally, I come with notifications (still) and motivation to make all that stuff working together (which is not, at the moment). This is a very, very interesting experiment since it kinda stresses ReactOS and forces me to work on ReactOS part I promised I would never work on (I am speaking of CC!). On another side, I will also keep on working on other parts of the kernel as I did previously, trying to improve it and match Windows 2003. I will also switch a bit on FreeLDR, some bugs are calling me there!

About personal projects, I have been quite active during 2010, even if I did not publish about them. One of the project I wanted to publish about before I forgot is a C++ garbage collector. I designed it for several uses, and finally it is more a memory manager than a garbage collector. Its purpose is simple: giving you memory whenever you need it and keeping track about it. It can also performs some operations on it to make your program debug easier such as: memory marking, memory zone tagging. It can also allocate non-paged memory, check against corruption, and so on. It has been designed to work in multi-threaded environment and provides functions for that. For example, when you share memory zones between threads, sometimes you even do not recall who is using what, how long. Here garbage collector becomes useful. Each threads when it uses a zone just needs to reference the memory zone. And once it is done, it dereferences it. Simple mechanism, but that ensure the memory zone will be released once every thread is done with it.
I am not totally done with that project (that is perhaps why I did not publish about it yet) and I plan to finish it and make it a bit closer to a garbage collector. And giving it the ability to allocate and release objects.

Other project I have been working on (and I am still actively working on) is an IRCd “new generation” written in C++. Its purpose is quite simple, implementing the five RFC concerning IRC, optionally adding extra often used/needed features. But, the new thing is that it comes with services implemented in (if built with, of course!). This is quite new, and interesting in my opinion. When you need to rapidly deploy an IRCd, configuring both IRCd, services (when you found the good ones!) can be a pain. With that IRCd, everything comes in. Thus it makes services really efficient as they directly communicate with the server (in a proper way, nothing messy!). And there is no need for SVSMODE, SVSJOIN, etc, commands or equivalents, here you just use services. At the moment, the core IRCd is almost complete and works really well. Services are mostly non existant (excepted OpServ, obviously).
For 2011, I plan to finish that IRCd, and perhaps to use it on heisspiter.net. Time will tell.

Finally, this is the shortened version, but there would be so much to say… Best thing is to keep reading ReactOS’ mailing-lists and this blog to keep informed!

Happy new year ;).

It’s been a long since I last wrote about ReactOS. This was mainly due to me being away from the project. Or, at least, passive. I was working on notifications, used by FSD (File System Driver). I finally ended my awayness period by writing a documentation about notifications, contenting all the information I gathered during my research about them. Then, I actively started working on them, coding them into ReactOS.

First of all, let’s quickly sum up what notifications are. To make it short, notifications purpose is to notify someone when a change occurs. In an OS, it means you can notify when a volume is mounted, when a file is added to a directory, when a directory is deleted, and so on. There are two ways to deal with those notifications in Windows/ReactOS. State notifications (mount, unmount, lock, unlock) are handled by PnP manager. On disk data changes are handled by both the FSD and FsRtl through a package of functions. So, how to use them? A client application (mostly in user-mode) registers a notification and waits for it to be complete. That’s that easy. Internally, things are getting harder, but the system are equivalent. For state notifications, PnP manager maintain a list of registered notifications. Each time a state change occurs, the driver (or even the kernel) responsible for that change has to call a PnP function which is IoReportTargetDeviceChange() or its asynchronous implementation: IoReportTargetDeviceChangeAsynchronous(). For FSD, there’s even an easier way to notify which is calling FsRtlNotifyVolumeEvent() giving the event that occured. FsRtl will do all the needed stuff and call PnP manager. Once PnP manager is called with a reported change, it just browses the notifications list, finds those that matches the report, and complete them. Caller is then informed about the change. What about on disk changes now? It works the same, just replace PnP with FsRtl. FsRtl maintains the list on FSD demand, browse notifications, takes reports and so on. To register a notification, FSD just calls FsRtlNotifyFullChangeDirectory(), and to report a change: FsRtlNotifyFullReportChange(). It doesn’t have to do more.

So, now, what’s present in ReactOS, and what’s not?

That question is hard to answer. But, let’s try to make the answer as clear as possible.

• About state changes notifications, technically, everything’s present in ReactOS kernel since revision r47837. It isn’t implemented as it’s in Windows, but that’s already a good begin for having such notifications. I also added some reports. For example, FAT driver reports when it successfully mounts a volume. But, in fact, nothing works. For the simple reason that, IoReport* functions need a PDO (Physical Device Object) to perform the report. Or, Windows (and ReactOS as well) calls the drivers to get the given PDO. In fact, you have a stack of drivers. Higher level is FSD, and lower level is the one that communicates with disk. So, to get the PDO, Windows calls the higher level driver, to ask for relations, and PDO. Then, drivers pass the request to the driver lower they know about in the stack until the last one which complete the request, giving the PDO. It goes down the stack. This side of notifications isn’t implemented as it’s done using PNP requests, and our drivers doesn’t handle them. I recently implemented that support to our FAT driver (r48560), but as the rest of the drivers doesn’t handle it, it does fix nothing. Work will have to be done!
• About on disk changes notifications, that’s the exact contrary. They work, but aren’t implemented. That could look a bit weird, but that’s not. In fact, those notifications are working in my working copy, but I didn’t release them yet, to have time fixing them. You may have already seen the screens I published about them: http://pierre.heisspiter.net/rostests/notifications.png & http://pierre.heisspiter.net/rostests/notifications3.png. First was the earliest test I did that worked. Full of bugs, of leakages, but it worked. In fact, it’s showing a Microsoft applications designed to test notifications (why using something else? :P) that you can find here. The way it works is easy. You start it giving a directory, and it will register notification on that directory for files changes. And a notification in root directory, for directories changes. So, on first screenshot, you see the application monitoring C:\ drive, and me saving a new file to C:\ called newfile.txt. The application successfully got the report. But given the code, that time, I was really lucky it worked! Second screen comes later, and shows the both notifications working. Saving two files, creating a directory, and the applications (same app started twice with two different directories) printing about both. There, code was getting cleaner and cleaner. Now, code is in a consolidation state. Which means I’m trying to make it rock solid, and I also try to understand details I still don’t get. I hope I may commit that code soon. First, notifications won’t be as complete as in Windows, but it’s a good step into having them. Only issue is that nothing is using them on ReactOS. No application (even not explorer) is registering notifications, and no driver is reporting changes (ie FastFAT/CDFS). So, something will have to be done there as well.

Now, what else? After having fought to get a trunk freeze, I switched to fixing ReactOS instead of keeping on working on notifications. As you might have noticed if you follow the project, the OS recently regressed to a state ever reached. Only a few applications are working, having ReactOS booting is hard, it appears those are due to some memory corruption, deep in the OS. Knowing the origin of such corruptions is quite hard. Indeed, recently, MM (Memory Manager) has been rewritten, and made less permissive than the version we had previously. So, we’ve got three choices: or new MM is broken, or rewrite throw some light on defecting ReactOS components (even in the kernel), or… both! Personally, I tend to believe that the last solution is ours. Even if that sucks. Such revisions show that MM is responsible for a part. Now, what? All the rest is OK? I don’t think so, either think Aleksey Bragin, coordinator of the project, and developer as well. We both agree after short talk that our memory corrupter could be FSD. Indeed, those have been designed and coded in the early ages of ReactOS, when kernel was poor. And then, they have been more hacked than improved to take advantage of kernel new features. I would even go farther actually. Saying that FSD and kernel are both responsible for that status. FSD drivers for calling the kernel with bad/broken parameters and the kernel for lacking proper checks. I’m thinking to a particular part of the kernel here. I’m talking about CC (Cache Controller). This part is involved in caching data, delaying read/write to the disk. So, it’s a heavy MM client. Our current implementation is really poor due to the complexity of the caching process. Checks are low. So, this plus broken driver can make huge damages. I think that’s the situation we’re in. Time will tell. I’m currently trying to track any (really, many are… Only a bit!) broken call to CC by our FSD that might have side effects. And I’m as well thinking about a hackish solution that may be found. For whatever reason, my working copy, with all the changes to FSD and kernel it has (and even to user-mode components!) is largely more stable than trunk. Why? I don’t know. But an hackish solution could be found in, just to help releasing and pushing fastfat_new to replace our current old fastfat driver. But, I’ll talk about fastfat_new and all the projects we (Aleksey and I) have about it in another entry.

Just keep in touch with the project, we’ll keep impressing you, in spite of those… blockers! Once those will be fixed, next release will be great. Really.

Recently, for one of the projets I’m working on, I’ve been asked a really naive question: “How can I be sure a function pointer is valid?”. If the question is naive, the answer isn’t that easy to find. In most programs, when pointers are checked, we just check whether the pointer is null. If not, we assume the pointer is valid. If that test is sufficient for most cases, it happens sometimes it’s not enough.

So my goal here was to find a way to catch wrong non null pointers. I immediately thought about huge and crappy solutions where pointer checks would have been slower than crashing and restarting.
Then, close to the end, you do stupid stuff, such as typing: man end. Here was the solution. 3 external variables are provided by ld when linking a program, and defined by loader when a program is started: etext, edata, end.
Let’s switch back to the structure of a binary. When you build a program, using GCC and without playing with sections, your program is cut into 3 parts: .text, .data, .bss. Interesting part for us is .text. This is were program code is stored. So, when you’re using a function pointer, its address will point into .text section, or it’s not valid. etext meaning end of text (section) is the address of the first instruction after the text section, so every pointer address has to be lower than etext. This gives the first way to check a function pointer. Then, I wondered: pointer has to be higher than something. What? And how to know. What was easy to find: base address of the binary in memory. Indeed, when you start a program, its code is stored in memory. So, I had to find the address of the first instruction. How? While browsing the web, I found that another symbol was also provided by ld: start. This is the address of the entrypoint of the program in memory (most of the time, the main() function of a program is the entrypoint). And most of the time, OS when loading a program puts entrypoint at the begin of the text section in memory.
So, I wrote the following function:
int is_fct_ptr_valid(void *p)
{
  extern char _etext, _start;
  return (((char*) p < &_etext) && ((char*) p > &_start));
}
This way, you can check in a more accurate way if a pointer function is valid.

Then, another question raised in my brain: “OK, you can check functions pointers in a nice way, what about memory now?”. Memory is something harder to check, and I was again a bit lost about how to proceed. I found a first way: most of the time, memory of a program is stored at the end of the program representation in memory. So, pointer address has be higher than end (cf: previous paragraph). And, in fact, having higher address is easy. You just have to use sbrk(0). sbrk is the function you can use to increment heap size of your program, it takes the size of the increment in parameter. 0 means no increment, so it just returns the current higher address.
So, I implemented that and tested. But it failed. I was really stupid thinking that would always work. In fact, in a program, you’ve got two kinds of memory: heap and stack. The method described above is only good at checking heap. Now, the question was: how to check stack then? There was no direct method way to check it, as it was possible for all the rest. Then, I thought about something a bit tricking. When you call a function, memory for local variables is allocated from stack. In fact, you just substract size you need from SP (Stack Pointer) and then, use stack past that given SP. And you do it each time you call a function, even inside another function. And main() is the main function. Then, if you know the address of the stack in main, you know that every other pointer has to be lower than it. And when you call the function to check memory, you know that’s the last called function, so that the pointer to check has to be higher than a pointer you would have in the function. The tricky method was born.
char * sstack;
int is_mem_ptr_valid(void *p)
{
  char estack = 0;
  extern char _end;
  return (((((char*) p > &_end) && (p < sbrk(0))) || (((char*) p < sstack) && ((char*) p > &estack))));
}
int main()
{
  char start_stack;
  sstack = &start_stack;
  /* … */
}
Here, you have the whole process to check memory.

Now, the test program, to show you the whole process:

/* Pointers checkings example */
/* Author: Pierre Schweitzer */

  return 0;
}

The functions given above, even if they are more accurate, are not fail-proof. Several assertions have been made because they are true in most cases. But, in case you don’t have contiguous memory, in case entrypoint isn’t at the begin of memory, those functions would be senseless. Furthermore, those functions only check if pointer points to a valid memory zone, not if the content are valid. Your program can still have issue due to of-by-one mistakes or such.
But, that’s a nice way to begin!