Situation of the file systems in ReactOS

Disclaimer: this was originally written as a reply to a topic on the ReactOS boards, regarding the GSoC projet that targets at booting ReactOS from BtrFS volume. This was an opportunity to give more details about the file systems situation in ReactOS, as it is often discussed among the community. One of the common question being: why do we have BtrFS and not NTFS? You’ll find the answer here!

I often read on boards people having strong opinions about file systems situation in ReactOS. About current situation, about ideal situation, about future situation.

So, let me try to clarify once for all current file systems situation, current goals, what are the under going works. In short: WHY is it that way and not another way. You’ll understand better why this GSoC project makes sense, not only to (attempt to) gain a new ReactOS developer, but also to improve our project.

Historically, from users point of view, ReactOS had two file systems: CDFS & vFAT. Both having their dedicated drivers. These drivers have been developed in ReactOS, against the ReactOS kernel and our libs. These drivers weren’t working in Windows and Windows drivers were not working in ReactOS. Both ReactOS drivers and ReactOS kernel have been made to work together. Our drivers need our kernel, and our kernel needs our drivers. This is clearly not the best way to improve ReactOS compatibility nor to make it progress.
There has been an attempt a few years ago (or perhaps a decade ago?) to develop a new FAT driver, in W2K3, using an external lib, and to fix ReactOS around so that we would be closer to what Windows kernel does. Unfortunately, because ReactOS is a FOSS project, the developer attention got drawn elsewhere and this driver got abandoned and then dropped. Our kernel didn’t benefit from the attempt.
In parallel, several developers were doing their best to improve our vFAT driver (which even became a FastFAT driver in the process ;-)), but still against ReactOS, which didn’t help regarding compatibility.

Some did the test, and just took the drivers from Windows 2003 and put them in ReactOS, it just BSOD and didn’t work.

So, how to improve compatibility ? Make our drivers work in Windows 2003, or make drivers that work in W2K3 work in ReactOS. First option is no go for me. I have no way to instrument a W2K3 (reminder, I develop on Linux ;-)) to have proper debug, so, only remains the second option.
We could have used MS drivers for this. As FSD implementation samples, MS provides CDFS and FastFAT driver. But two problems raise: first one is a half driver (only RO), and second is really complex. Hence, they are good examples, but not that easy to debug.
Then comes 3rd party drivers: ext2fsd, winbtrfs, and so on. These work on Windows 2003 (or nearly ;-)) and can provide good test case. And this is the choice I made. Getting these in ReactOS. It has two benefits: providing live test cases for our kernel and bringing in new features. Sounds great, isn’t it? And it worked. We brought support for these FS, but we also squashed a number of bugs in our kernel at each release of these drivers breaking in ReactOS. Our kernel started working better. For the record, Trevor Thompson, a GSoC student, even helped on a part.

Thanks to these efforts, we came to the point where we could drop our own CDFS implementation in favor of the Microsoft one. It just works out of the box (or nearly ;-)) in ReactOS.
Nowadays, our kernel isn’t still ready to handle setup with MS FastFAT, but could work once installed with it. At some point, ext2fsd and MS FastFAT seem to expose the same issues when it comes to setup. Fixing one should help the second one. BtrFS could be already used to install ReactOS. It requires less fixing, and less complicated fixing, no on-disk corruption, no broken behavior in the kernel. It is then more interesting for a student to work on it and bring ReactOS on it; while being less frustrating for the student, and more relevant for the community as regular progress can be shown. It will pave the way for fixing the other FSDs. Every single bit added in the right direction is a plus for any FSD. It’s definitely not wasted time. And it will also help ReactOS in general: there are software, features, APIs we cannot test because FAT is a too limited file system (for instance, no hard links). Having BtrFS setup will allow testing these code paths without requiring second hard drive and so on.

Now that’s said, what about NTFS? NTFS support in ReactOS is a complex topic. It was the first thing I worked on when I joined ReactOS (remember, in 2007 ;-)), then I moved to other topics. I later came back to it, and brought it to a level were it could read from any NTFS partition (or nearly ;-)). We had the chance to have a student working on it for two GSoCs: Trevor Thompson. He even brought in initial write support. Unfortunately, our NTFS driver development started along with vFAT and CDFS drivers and suffers from the same issues. It is really ReactOS specific. I understand NTFS is important for ReactOS, but it requires hard work to complete, and waiting for it to fix our kernel compatibility would be a mistake that would make the project stay behind. So, it will come, one day. It just needs more time to mature. And no, NTFS-3G is no use here (nearly ;-)). It is a usermode implementation for FUSE that cannot be just translated in our kernel. And no, you don’t want to install ReactOS on our NTFS driver, it would be slow as hell, it doesn’t support any caching, yet.

I hope you understand better what’s going on regarding file systems in ReactOS and will be less prone to say this GSoC slot is misused ;-).

Rome blabla.

Aspect Oriented Programming using C++11

Now that G++ fully supports C++11, it is time to look at some fancy feature which has been usable with C++11: Aspect Oriented Programming (AOP). The purpose of AOP is to make code even more modular. While you have core code, you can modulate its behavior and complete it by using aspects.

So, basically, one of the idea (taken for AspectJ – for Java) is to encapsulate functions to add extra behavior (called “advice”). Key feature in C++11 to implement this are variadic templates. Let’s imagine you want a dummy advice that simply displays all the arguments passed to a function. The idea will be to implement a generic function, using variadic templates that will wrap calls to any function and display their arguments.

Basically this is it:

template<typename FunctionType, typename… Args>
auto AspectDisplayArgs(FunctionType * Function, const Args&… FunctionArgs) -> decltype(Function(FunctionArgs…))
{
  AspectpDisplayArgs(0, FunctionArgs…);
  return Function(FunctionArgs…);
}

The internal AspectpDisplayArgs() function will effectively go through the variadic template recursion and display args one after the other.

It will look like that:

template<typename Arg, typename… Args>
void AspectpDisplayArgs(unsigned int Count, const Arg& FunctionArg, const Args& … FunctionArgs)
{
  std::cout << “Arg ” << Count << “: ” << FunctionArg << std::endl;
  AspectpDisplayArgs(++Count, FunctionArgs…);
}
void AspectpDisplayArgs(unsigned int Count)
{
  // Do nothing
}

You can see that it is rather simple. It only displays the argument and calls itself to display the next one. Template specialization is here to stop recursion on last argument.

How to define join points? Here, no pointcuts available, so we’re back to call wrapping. Let’s say we have a function with the following prototype:

int Add(int a, int b);

Then, call it with the aspect would simply be:

Res = AspectDisplayArgs(&Add, 9, 11);

Rather easy! This wouldn’t have been possible with C++ < 11 because of the lack of variadic templates (and macros actually). You would have to write a function with template matching the prototype to the function on which apply the advice. So, not really useful.

Continue reading “Aspect Oriented Programming using C++11”

Using VMware Player with libvirt

In order to catch any regression in ReactOS, but also to develop ReactOS using test-driven development method, ReactOS devs were using two testbots. One was running all the tests in a ReactOS virtual machine run by KVM under Linux, while the second one was running them in a ReactOS virtual machine run by Virtual Box under Windows.

The fact is that, under Linux, to achieve such work, we are using an application developed internally named sysreg2. This one, uses the libvirt API to create a KVM domain, to start it, and to check the serial output of the VM, to get the results of the tests run in ReactOS.

We recently decided that another testbot could be handy. It has been decided that as a first step, using VMware Player under Linux would provide interesting results. In order to keep the behavior of the testbots unified, sysreg2 was to be adapted to also manage this VM.

Fortunately, libvirt also provides a compatibility layer with VMware Player. The API is still the same, so, normally, it would not have required that many efforts to get the switch, as long as, the good connection address is provided for libvirt to the hypervisor: vmwareplayer:///session, in our case. Then, internally, libvirt simply convert its XML domains to VMX domains and uses VMware tools: vmplayer and vmrun.

But actually, it did not work that easily. First, VMware Player tools were silently failing at startup. A simple strace shown that the tools were trying to load several X11 libraries. And when those had been installed, the tools were lacking a DISPLAY. Installing a complete X11 server and client just for VMware would have been kind of overkill method.

The solution used has been to make usage of two bootstrap scripts for VMware tools and of XVFB. XVFB is an X virtual frame buffer. So that, the applications run within this VFB just think and act as if they had a display, whereas they actually do not really have. So, once installed, the bootstrap scripts were basically just issuing a xvfb-run of the VMware tool with its args.

Expect for vmrun. Two other issues raised. First, it needed a complete X11 install along with GTK to be able to display the VM window. The solution here has been to append, at the end of the run command the “nogui” instruction. So that vmrun does not try to display anything. The bootstrap script was modified that way, as we cannot really manage the libvirt commands issued to those tools. The other issues also came from libvirt commands. When libvirt tries to destroy a domain, it asks VMware to do a “soft” shutdown of the machine. In case (and it happens with ReactOS…) where the machine is totally locked, this is pointless. There, the bootstrap script has been modified again to force to “hard” any shutdown instruction it was receiving before passing to the real tool.

Also, VMware and KVM do not exactly support same kind of hardware, especially for serial devices. So, it has required rethink and rework about the way sysreg2 and VMware can communicate. Finally, the adopted solution was to make sysreg2 create an Unix socket so that virtual machine can connect on it and write (also read).

We got a testbot working successfully :-).

Continue reading “Using VMware Player with libvirt”

Experimenting AOP with code profiling

During my studies at the ISIMA, I have been quickly presented MDE (Model Driven Engineering). We went through AOP (Aspect Oriented Programming), which is somehow a superset of OOP (Object Oriented Programming). With your objects, you can describe aspects and this will help you write simpler code, and more easily maintenable code. It also allows cutting in a clear way the application into modules. Some modules being generated during the complete build of the project.

So, with AOP, you can define points (known as pointcuts) in code where you will add other code, developed in parallel, responding to an aspect. This allow to do transerval checks or treatment to the application. Needless to say that it is really useful for applications security and stability. This is where the idea of code profiling is born.

It is possible with AOP to define pointcuts at the begin of a function execution and at the end of its execution. So, if you are able to match any function, it is virtually possible to add a pointcut to any function begin and end. Furthermore, once you know when a function starts and ends you can calculate how much time it needs.

This is what I do in my simple (for the moment?) framework. It just consists in an header for AspectC++ that will define two pointcuts: one before the begin of the execution of any function (be it a function for a class, an operator, or whatever) and at the end of the execution. Then, using the Linux system call time, it will get the time at which those operations occurs to gather data. The time resolution is low for the moment, the only purpose being to validate the concept. Once it is done with the execution of the function, it will store all the data into a structure remembering number of calls and total length for all the calls. This does not provide that many information for the moment, but it has proven to be working.

The strength of such approach is that it makes the whole code generic. There is no need to modify the code that has to be profiled to get it profiled. It just need to be parsed with the profiling header and then compiled. The code itself for profiling is pretty simple for the moment (as its output).

An example of execution works as following:

$ ./main
You want number of primes lower than? 999999
Result: 78498
Function: int main(int,char **), calls: 1, total time: 6
Function: unsigned int FrequencyOfPrimes(unsigned int), calls: 1, total time: 3

This is still work in progress for the moment, and the idea would be to get more features like better timer resolution, real support for recursive calls. But it keeps being a good beginning to show what are the features offered by the AOP.

You’ll be able to find the example code on GitHub.com.

IPv6 Launch Day at the ReactOS Foundation

On the 6th of June 2012, the Internet Society has decided to start an important and world wide event: the IPv6 Launch. This means that on this days, several Internet actors (website editors, network operators, hardware ventors) will make what they own IPv6 compatible (in a transparent way!). And will ensure it will stay that way. This is not the IPv6 day.

Regarding Internet then, this is a major event. And an agreement to get IPv6 launch is a great decision. This is why we [The ReactOS Foundation] decided to associate our name to this event. This means that several of the ReactOS servers will be compatible with IPv6 (not the OS itself!). A lot of work will have to be done (even if it was already done partly previously) to ensure everything gets ready on the 6th of June. A kind of roadmap is available here: IPv6.

This actually requires work. Not only to update configurations (this could be the easier part…). The real problems come from software that don’t support IPv6 properly and that need a rewrite or even worse a full rewrite. This is what I will be working on up to the 6th of June to get everything fully functional.

As an example, in order to “help” DenyHosts (which doesn’t handle IPv6 even with custom regexp), a new tool has been developed to run nearby it. It will catch all the non-allowed connections from an IPv6 address to SSH and will drop the IP in /etc/hosts.deny. All the tools that may be developed will be dropped at GitHub under GPL licence (ForbidHostsv6 is already there, follow the link). To permit everyone to make use of them, in case they also need them for the switch.

Let’s share our knowledge about that switch, to make it as easier as possible (and the smoother possible as well!).

Nagios plugins

All the servers we have at the ReactOS Foundation are monitored with Icinga (a Nagios fork). Icinga is compatible with both nagios plugins and NRPE (Nagios Remote Plugin Executor). If Nagios plugins are enough to monitor basic services on our servers, with specific services and value to monitor, those are not covering enough. So, as many system administrators (I believe), I went to Nagios exchange to get more plugins to have those matching up our needs.

First problem is that with those plugins, most of the time you have overkill plugins, that do much more than what is required. They also not necessarily match what you need (best example is our hardware raid, I had to modify a script from nagios-exchange).

Second major problem is that those plugins are most of the time (not to say all the time) scripts written either in bash/python/perl. If that’s suitable for small monitoring, it comes to be a problem with larger solutions (~1000 nodes).

Even if at the Foundation we don’t have 1000+ nodes (yet? – unbelievable), I really wanted we had native plugins, written in either C/C++ to match specifically our needs (and kick out some problems). This lead to the development of plugins. While our internal scripts plugins were private, I decided to release those to the outside to allow everyone having the same issue to enjoy those.

Those always match name and behavior (arguments and return – especially in performance data) of the previous scripts we were using.

For the moment, only check_ram (100% internal one), check_acpupsd, and check_mountpoints are currently available (the only ones to have been developed yet). More are to come.

You can find them at: http://svn.reactos.org/svn/project-tools/trunk/nagios/

Feel free to comment or to sent patches to improve them!

ReactOS plans for 2012

This year, instead of looking back on previous year to sum up what has been done, I will speak about what is planned.

First thing is the long awaited 0.3.14 release. It is on the way to be released and has been branched. This will be an important ReactOS release. We took time to prepare it, and it comes with numerous and great changes. Unfortunately, as several rewrites took place, some regressions are to be expected. And some people may not be able to install ReactOS any longer. In case it happens (in VMs), change your virtualisation software (or downgrade it in case it is Virtual Box). We know that is a real problem. But in spite of our efforts, we could not kick it out. It will require huge work. But, as expressed there are ways to work around.

Most of my work on ReactOS will take place in background. In 2011, I became one of the ReactOS systems administrators. Which means, ensuring servers are running properly and up to date, but also deploy new things to make developers everyday life easier. This year, we plan to deploy two more test bots. One on VMware ESX, another on VMware Player on Linux. Both handled through libvirt and our sysreg tool. I already started implementing the support in our tool.

Another thing will be the (also long awaited) switch to CMake. We will drop support for our own solution (rbuild) to use CMake to handle ReactOS builds. This has been postponed until release and until ReactOS Build Environments (RosBEs) are ready. Both are about to be completed. On RosBE-Unix, I took over on Colin, as he has less time to give to ReactOS. We also have a last build slave running rbuild builds. Once CMake switch will be done, it will be time to tell him good bye. It will be a real switch for the project. The last rbuild builder has done more than 10k builds!

I also plan to deploy another static analysis tool on our server to check ReactOS code quality. We already have cppcheck (with quite huge configuration, by the way…) that returns pretty good results. We also make use of Coverity with success. The idea would be to add another and rather different tool: sixgill. This tool appears to be successfully used by the Mozilla Foundation.

The Foundation also have a Windows Server 2003 license that we would like to use to set up a test environment for developers. That way, they could write tests for functions, run them on Windows 2003, and implement/fix the right way on ReactOS. Method which is also called: test-driven development. Rather useful for us!

Finally, if I have still a bit of time (looks difficult!), I would like to finish all the things I have started/planned last year on ReactOS. But I guess it will not be that easy…

ReactOS & GSoC (and all the rest…)

The 18th of March has been quite a great day for the ReactOS project. Indeed, on that day, Google released the list of the accepted mentoring organisation for the Summer of Code. And for 2011, ReactOS is in. Last time the ReactOS project was accepted was in 2006. So, this has been something quite magic for the project.

But, you may wonder: what are Google Summer of Code? The idea is simple. Each year, Google will select organisations (ie, Free Open Source Software projects) based on applications they send. Then, those organisations, called mentoring applications, will be able to ask “slots”. Those slots are students that the organisation will mentor on a project, an idea. The important part is that, to ensure students will come (and organisations will apply), Google is offering $5500 per completely finished mission ($5000 go to the student, and $500 to the organisation). That way, students who don’t have any internship can earn money during the summer, and FOSS organisations can earn money as well.

This year, the ReactOS will host 5 slots. Which means that we will welcome 5 five students. You can find some ideas of projects to work on here. But students are free to purpose another subject when sending an application. Students can’t apply yet and have to wait until the 28th of March (before, they have to learn a bit about the projects).

Mentors you may except to have on ReactOS aren’t known yet. But, you’re likely to find Aleksey Bragin in the list (it would be weird if he wasn’t in!). Actually, list will be known once students and projects will have been chosen. This is done that way to ensure that mentors will match at best subjects (and prevent any issue with the mentor). Anyway, several ReactOS developers already applied to be mentor. So no shortage of mentors in sight!

Another topic regarding ReactOS: I’ve been presenting ReactOS at the ISIMA on the 17th of February. Part of the presentation has been shot and you can find the uploaded video here. I may do another ReactOS presentation in France this year, but nothing has been confirmed yet. So, follow the ReactOS news for the moment!

I hope you’ll enjoy the video and the information given in. If you want to get the PDF used for the ReactOS presentation, you can find them here.

That’s all for the moment (nothing related to development yet). More will follow (about dev & GSoC). And don’t hesitate to join the ReactOS project. GSoC are a great opportunity!

Let’s go back to 2010

Indeed, now 2010 is over, it may be interesting to go back to it. Not for the same reasons that led the ReactOS to go back to 2010 (cf: r50529, a revert needed due to a bug on boot image handling). In fact, it is interesting to go back to 2010, just to see what has been done, what was planned, what has failed. Finally, what we can say and remember about that year.

First of all, I would like to speak about my servers. As you may know (or not) I currently own two servers, known as www.heisspiter.net and www2.heisspiter.net, and I also rent a third server at OVH, called www3.heisspiter.net. I was not speaking about them, because I was not really taking care of them. And this led to many and important issues (bad performances, bots, and so on). During a weekend, I decided to switch www3 to ipv6, which finally worked. But, it made me understand that my servers needed love, somehow. So, now, I am working back on them. The idea that they have reached some maturity point and can work alone is definitely wrong.  Indeed, 2009 was already a pretty bad year for heisspiter.net and 2010 was terrible. No evolution, several issues, servers down, … A quick look at statistics show that people also stop coming on the server. And I cannot blame them. 2010 was a really bad year for heisspiter.net, and 2011 cannot be worse. I will just do my best not to make it the same.
An encouraging note regarding heisspiter.net, at least. Some evolutions have started. I was talking about ipv6, but also a webmail for users arrived, upload service is back, mail server has been fixed, servers software have been updated, and heisspiter.net internal tools fixed. This actually explains why heisspiter.net is more stable since December!

Other major point… Of course, ReactOS! You may have seen, reading my other posts that the project did important step into stability and features. Some rewrites, some parts becoming more mature. MM rewrite, with Heap rewrite (for user-mode land) force developers to fix the ReactOS code to corrupt less memory (or to less corrupt memory?). And this works. Especially when fixes are applied to those rewrites. This also comes after some hard year for ReactOS, with no releases, and nothing to release, due to broken trunk. But, here, it is past!
My modest goal, for 2011, is to prove that ReactOS has gained some maturity now. Some testers are already pushing to get 0.4, and I would like to show we are not that far. And I will try to show it on my domain of work on ReactOS, ie filesystems and kernel. With the help of Johannes Anderwald, and Art Yerkes, I attempt to make ReactOS boot from Microsoft FastFAT driver. Johannes has brought some code for tunnels handling in FsRtl, Art code for MCBs and CC. Finally, I come with notifications (still) and motivation to make all that stuff working together (which is not, at the moment). This is a very, very interesting experiment since it kinda stresses ReactOS and forces me to work on ReactOS part I promised I would never work on (I am speaking of CC!). On another side, I will also keep on working on other parts of the kernel as I did previously, trying to improve it and match Windows 2003. I will also switch a bit on FreeLDR, some bugs are calling me there!

About personal projects, I have been quite active during 2010, even if I did not publish about them. One of the project I wanted to publish about before I forgot is a C++ garbage collector. I designed it for several uses, and finally it is more a memory manager than a garbage collector. Its purpose is simple: giving you memory whenever you need it and keeping track about it. It can also performs some operations on it to make your program debug easier such as: memory marking, memory zone tagging. It can also allocate non-paged memory, check against corruption, and so on. It has been designed to work in multi-threaded environment and provides functions for that. For example, when you share memory zones between threads, sometimes you even do not recall who is using what, how long. Here garbage collector becomes useful. Each threads when it uses a zone just needs to reference the memory zone. And once it is done, it dereferences it. Simple mechanism, but that ensure the memory zone will be released once every thread is done with it.
I am not totally done with that project (that is perhaps why I did not publish about it yet) and I plan to finish it and make it a bit closer to a garbage collector. And giving it the ability to allocate and release objects.

Other project I have been working on (and I am still actively working on) is an IRCd “new generation” written in C++. Its purpose is quite simple, implementing the five RFC concerning IRC, optionally adding extra often used/needed features. But, the new thing is that it comes with services implemented in (if built with, of course!). This is quite new, and interesting in my opinion. When you need to rapidly deploy an IRCd, configuring both IRCd, services (when you found the good ones!) can be a pain. With that IRCd, everything comes in. Thus it makes services really efficient as they directly communicate with the server (in a proper way, nothing messy!). And there is no need for SVSMODE, SVSJOIN, etc, commands or equivalents, here you just use services. At the moment, the core IRCd is almost complete and works really well. Services are mostly non existant (excepted OpServ, obviously).
For 2011, I plan to finish that IRCd, and perhaps to use it on heisspiter.net. Time will tell.

Finally, this is the shortened version, but there would be so much to say… Best thing is to keep reading ReactOS’ mailing-lists and this blog to keep informed!

Happy new year ;).