Posts Tagged ‘Google Summer of Code’

GSoC Wrap-Up Review

Hello!

This year’s summer has just come to and end, and so has the Summer of Code. It’s time to go over what I did while working on a performance testing framework for DUNE.

First, I wrote a Python program that measures the run time and resource consumption of an external program. It stores quite a lot of data, but the most useful are definitely time spent, top memory consumption, and computer parameters such as number of CPUs. The tool can then output this data into a temporary log file, store into a sqlite database, or upload it to a central server. Furthermore, it can generate nice graphical representations of the data in the form of HTML pages with javascript graphs. It is written in a very modular way, so while there is a script to tie it all together, each of the described actions can be done separately. This also minimizes external dependencies, so if the user doesn’t have the Python SQLite module installed, the database part is skipped.

Then, with help from my mentors, I tied the measurement tool into the DUNE build system, or rather, both of DUNE’s currently available build systems, autotools and cmake. This allows you to set minimal configuration, and then just run “make perftest” from the build directory, and all performance tests are performed, measured, stored, uplodaded and visualized. By using the build system directly, we can get information about the compiler used and its flags. This is important if you ever want to compare compilation with different compiler options. When run this way, the tool separately measures both the compilation time and run time. The compilation time may get quite long with a lot of templates, unnecessary includes and in general large code files, so such a test comes in handy for identifying compilation bottlenecks.

For displaying the results, I used Twitter Bootstrap, Dygraphs and Table.js, so the generated pages look quite nice. Graphs are interactive, and some table columns can be filtered for easier browsing. Some examples are shown here.

Results of a single text run

Results of a single text run

final-vis-cr

Graphical representation of results of repeated runs of the same test

Finally, I added a server component, implemented as a number of CGI scripts in Python. One of these endpoints receives uploaded log files, while another stores them into a ‘proper’, PostgreSQL database. This two-step process is used, so that processing can be done in batches and separately from uploading. For example, one could easily upload the data in some other way, like the secure copying with SSH. The current uploading setup is not completely open, as it requires a username and password, but these are stored in plaintext on the server.

With a server that accepts data from multiple computers, I could add some additional views. For example, there is a page that identifies outliers with adjustable tolerance. Outliers are data points with considerable deviation from the mean, which in our cases means unusually long run or compile times.

Server overview of all collected data

Server overview of all collected data

Results of a single run on a server

Results of a single run on a server

Aggregated results of a single test on a server

Aggregated results of a single test on a server

A page for finding outliers on a server

A page for finding outliers on a server

All in all, I would say my summer project was a success. It started a little slow, at first with a two-week pause because of a summer school I attended, and then when had to finish my master’s thesis a month before I expected. However, despite not always following the set schedule, I tried really hard to complete everything I set out in the timeline. This project had several different components, from the DUNE libraries in C++, the two build systems, Python, database programming and websites, so I had to learn some new things over the summer. For this I am grateful, and I would like to thank the DUNE developers again for giving me this chance.

I’m sorry I can’t attend the DUNE developer meeting this week, even though the developers invited me and even offered to cover my expenses. I’m giving a talk on the optics of liquid crystals at a conference in Kranjska Gora that happens to take place the exact same three days. However, I can say I enjoyed working on this project, and can only hope that my contributions will help others.

Thank you!

Dune mid-term review

Hello!

Last week was the mid-term review for the GSoC. Because of this, I spend more time polishing and completing existing things than adding new ones. The biggest was documentation, I added docstrings to all the functions I’ve written since the start of coding. This should hopefully make it easier for everyone else to see what I did, but more importantly to extend it in the future.

This was, however, not all. Dune-perftest now has a couple (= two) example programs, written in C++ using the DUNE libraries. One is mostly empty and basically just measures the time needed for MpiHelper initialization, while the other works with matrices. Such programs will be used for monitoring the performance of DUNE itself. In order to build these C++ programs, I had to use the DUNE build system, based on autotools. I probably spent far more time than I should have on this one. As mostly a KDE developer, I am only used to CMake. I know that DUNE already supports CMake, and if I understand it correctly a complete move is planned, but at the moment I will include both.

There are no new screenshots, because graphically nothing has change since the last post. The actual generation of templates is somewhat improved, and the page (and graph) only shows data for the same command. I’m pretty happy with how both Bootstrap and Dygraphs turned out, I will probably redesign the page a little, but the graphs look good enough to me. However, I will add more information, starting with the memory footprint.

Now that the first half is over, I have to start planning ahead. My short-term goals are more automation and some statistics. More automation means you should be able to test multiple programs with one command. A couple more example C++ would help a lot for testing this. I will also make it possible to define both compile and run commands and have those associated with the same program. DUNE is mostly a template library, and these can often cause very long compile times. Once testing is automatic enough, there will be more data, so a need for meaningful statistics will arise. These can be basic enough, identifying outliers and general trends will be my first priorities.

 

Becoming a KDeveloper

So, I’ve just been accepted into this year’s Summer of Code program. I’ll be working on KDevelop, introducing a system for using and sharing code templates. This includes both project templates (integrating KNewStuff into KAppTemplate) and more specialized templates (for code files,  classes, tests, make and cmake files, etc). The full proposal is online at http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/noughmad/37002.

However, all this is still in the future. I’ve already been doing some work on KDevelop, and so far I’ve enjoyed it very much. Not only do I use it every day, its plugin architecture makes it easy to add small pieces of functionality. Note that because of school getting harder every year, I have little free time and my progress is a bit slow. But anyway, I’ll show you what I wrote the past few moths.

Unit tests

KDevelop currently has no support for unit testing. However, both KDevelop and KDevPlatform have loads of them, so a good system for running tests and reporting their results would find a good use. So I introduced some new interfaces for unit tests, including finding and running them. A toolview was adapted from the old and unmaintained Veritas library, however I took steps to better separate the interfaces and their implementations. Currently supported are tests declared with CMake and those using the PHPUnit framework.

A toolview for listing unit test suites and their test cases

Apart from simply listing tests, the relevant declarations are also extracted and a “show source” action is available. Support for running unit tests and reporting results is also functional, however I have trouble deciding which icons to use. There is a difference between a failed test case and an error, and also between an individual testcase and suite results. As a short-term solution I implemented a mess of different icons.

The toolview showing results of a test run

Finding tests declared in CMake files works reliable, whether they use add_test() directly or some wrapper macro (like kde4_add_test()). Unfortunately, this is not true for php tests cases. The problem with PHPUnit is that a test case must inherit from a certain class, but  doesn’t have to include any special file, so the parser find the right declaration by itself. I managed to fix it by modifying the parser, so that most test cases are found, but unfortunately still not all.

Additionally, when running the test cases, their output is displayed in a separate toolview. For test cases using QtTest or PHPUnit, the output is highlighted appropriately (green for success, red for failure). For Qt-based tests, clicking a line in the output also takes you to the declaration in the code.

An output from a test using the QtTest framework

If you wish to test this, the code is in the “unittest” branches of kdevplatform, kdevelop and kdev-php. Any input is appreciated, especially regarding the presentation to the user.

Code checking

Another valuable resource for KDE developers is the Krazy code checker. Seeing how easy it is to create KDevelop plugins, I wrote another one to run Krazy and report back the results. Any issues found are inserted into the declaration-use chain, so they are visible in the Problems toolview, as well as in the code itself.

An issue found by Krazy displayed in the Problems view and in the code

The code is still in my personal scratch repository at http://quickgit.kde.org/index.php?p=scratch%2Fmihac%2Fkdev-krazy.git&a=summary

When all else fails

I also implemented a different kind of code checking: by humiliation. If you stumple upon a horrible piece of code, either written by you or inherited, you can earn a feeling of superiority by sending it over to The Daily WTF.

A context menu entry for submitting an offending code snippet to The Daily WTF

I haven’t yet submitted any code with it, but I tested the plugin with Alex, the site’s maintainer, and he says it works. So watch out, if you write bad code, even if you fool all the unit tests and code checking tools, someone can still expose you.