Yet another hardware review

So, I've not resisted the temptation and have bought yet another gadget.

Introduction.

I figured sometimes it's more comfortable to have a separate ebook reader than open up a laptop. Still, despite their power efficiency and weeks of battery life, I hate e-ink screens, they look really blurry. And most readers have ancient CPUs like Intel PXA270 (which was quite popular in 2006, btw). I didn't really feel like waiting a couple seconds till a single page of a PDF is rendered. So I went  for a tablet.

This time it's a Barres&Noble Nook HD+. I did have previous experience of having an Acer Iconia A500 tablet (which I later gave to my cousin for hacking). And I definitely had some important criteria when choosing the device:

• It should be lightweight. Not more than 600g. The 900g Iconia was barely possible to hold with one hand
• Screen should be 9'' or less. Otherwise, one needs two hands not only for typing, but for holding the device as well
• Screen resolution must be high. At least 200PPI, but 250+ is preferred. I really hate seing separate pixels (though I once had a phone with the 311PPI and still I could see them, but that's an extreme)

Unfortunately no nice Windows RT devices are available on the market. I mean, a 7 inch Windows tablet. Or take a look at those 10'' Asus tablets and MS Surface Pro. $1000 for something huge, heavy, ugly and with a 1366x768 screen? Microsoft has delivered nothing but a shipment of fail again, though no one was really expecting anything else.

So, why did I not go for the iPAD? Well, there were some reasons:

• iPAD 4 is huge. And it's quite pricey for a 10 inch tablet
• iPAD mini has an awful screen. 1024x768 is too low even for 7''.
• The OS is not very hacker-friendly. And being a computer geek I want to tinker with any piece of hard- and software that slips into my hands.

Hardware

Nook HD+ 's primary function really is a book reader, not a tablet. Therefore, it lacks most common tablet features. It has no:

• 3G
• Camera
• GPS

Actually, I think that's not bad. In fact, I've never used any of them on a tablet since I have a phone for that. Moreover, from the hacking prospective that means less messing around with drivers and proprietary binaries.

The screen resolution is 1920x1280 @ 9'' giving it the 256 PPI, slightly less than iPAD4 (which has 264), but quite nice anyway. The screen is nice, without any ripple or grid.

OMAP4470 is quite a fast dual-core CPU, and the user interface seems to be responsive, without freezes.

The downside is of course the proprietary USB connector and the lack of USB charging support from the PC (yeah, the stock charger is 2A, but USB 3.0 could in theory power that).

Software

Overall, the default firmware is quite nice if you're going to use the device as an ebook reader, not as a tablet. The Android UI is very well hidden behind the revamped graphics.

The most annoying thing is the welcome screen on the first boot that will not let you use the device unless you have the wifi connection to download the firmware updates and sign up for the B&N service.

The book reader app is quite nice and there are some free books in the store, which is a plus. Besides, you can of course copy the PDF files to the device manually.

The device doesn't have Google Apps installed by default - so no GMail and no Play Store
The lack of Play Store brings another interesting point. The majority of free apps from Play Store (like the Cut The Rope game) cost around $2.0 here. So just root it and install GApps :D

Hacking

The good news is that this device is very hackable and friendly for the free software developers.

First, let's look at the bootup process.
The OMAP4 System-on-Chip used in the device implements the secure booting by establishing the chain of trust.

Chain of trust

The chain of trust begins with the code in the SROM (secure read-only memory). This is an OTP (one-time programmable) area inside the CPU meaning that the code cannot be changed once written there. It then verifies the secondary bootloader using the public-key message authentication code. The secondary bootloader (SBL) in turn verifies the authenticity and integrity of the OS kernel. And then of course every OS has miriads of security holes like dangling pointers, lack of array bound checking and are vulnerable to stack smashing techiques. So if you want to hack the device, there's always some way  to exploit a vulnerability in linux. But even if a user can run their "malicious" untrusted code, the vendor can

When the chain of trust is implemented properly, the device is unhackable. Like motorola omap3 phones where the only means of running custom code is kexec from linux. Luckily, most vendors make stupid mistakes and the whole chain is easily compromised. For example, early revisions of the Samsung Galaxy Nexus phone had the xloader which did not check the SBL signature. Since the xloader was signed by Samsung, it allowed to replace SBL with a custom bootloader (u-boot) which I'v certainly done.On the nook this security measure is not used at all.


The bootloader on the device is locked, and doesn't seem to support fastboot.

The good thing is that the B&N cannot lock down the device in further software updates (only hardware revisions) because the chain of trust is disabled and the SROM code loads arbitrary code from the miniSD card. And this allows to run any kind of custom ROMs.

There already exists a CM10 port which converts the device into a full-featured tablet.

The only really non-free part is the PowerVR GPU. While there exist open-source drivers for some ARM GPUs (freedreno for Qualcomm chips even has a working mesa driver, while Mali in Samsung Exynos is only capable of running Quake 3 (linked against the driver) with the free Lima driver). For PowerVR there are unfortunately no free drivers but the closed-source ones exist for X11, are package for Ubuntu and should be quite good.

I guess I should try to get Ubuntu and Genode running on the tablet. Just for fun. Because we can :)

Read More

Visions on Genode OS development

Introduction

This documents summarizes my visions of what and how could and should be improved about the Genode Operating System Framework in order to make it usable on a day-to-day basis in various applications.

Please note that some of these opinions may be biased because my main interest in Genode is using it as a virtualization solution for embedded systems (namely, smartphones and tablet computers). Some of the ideas are inspired by an experience of porting Genode to a Samsung Galaxy Nexus phone, so they are correlated with what I'm planning to work on throughout 2013. Norman Feske of Genode Labs has expressed disagreement over some points here so you're welcome to comment and add your ideas and we'll see how it turns out.

And yeah, I want that any mobile phone geek and an embedded hacker out there reading this post joins the effort to make Genode running on at least one commercially available phone. Cmon, XDA crowd, where are you?

While this text may seem overly verbose, I tried to explain the points in a simple language so that my motivation behind each idea is clear to every reader, even those unfamiliar with the area of the Operating System development and without the knowledge of embedded hardware design.
Overall, I think we need to port a lot of functionality common to most OS kernels, like drivers, file system support and networking protocols, but at the same time I would like to avoid reimplementing the whole linux in C++. Perhaps the optimal solution is keeping drivers inside Genode, and letting a paravirtualized linux instance manage networking and serve as the platform for running userland applications.
In fact, Genode on ARM today is like linux was 7-10 years ago. Indeed, you can make it run, but the device driver infrastructure is missing and you'll have to write everything from scratch. And it will be slow and power-hungry.

Security

Currently, the strongest security issue with Genode is the lack of control over mapping the IO memory. Any service having the capability for an IO or connection is able to map arbitrary memory area. Besides, memory mapping is exclusive and multiple services cannot map the same region. This brings both security issues and programming inconvenience.

IO

I propose that a new IO memory mapper be written to provide coarse-grained region mapping. Firstly, it should be initialized with a list of memory regions. Each region should contain its name, physical base and length. Here's an example of how the config would look like:

<start name="New_IO">
 <binary name="New_IO"/>
  <config>
   <region name="I2C_1" start="0x40100000" size="0x1000" />
  </config>
  <provides><service name="IO"/></provides>
</start> 

Each service requiring RM access should use a proxy to limit its access to specific regions. Example:

<start name="IO_limiter_clk">
 <binary name="IO_limiter"/>
  <config>
   <region name="ClockControl"/>
   <region name="PowerDomain"/>
  </config>
 <provides><service name="IO"/></provides>
<start> 

For some SoC (System on Chip, essentially a microprocessor and the logic that comes inside of it like display or memory controllers) IO regions, we could define them inside the service code (platform_drv driver) so that definitions don't have to be duplicated inside the XML configuration file.
Such an approach involving looking up the regions by name eliminates the need to store the region definitions in header files and duplicated them in configs for proxies.
Alternatively, we could write a proxy that would restrict IO access to a specific memory area and create an instance of it for every service that wants to access IO memory. The downside of this approach is that IO region definitions will be scattered around the whole config file and possibly duplicated in both C++ headers and configs.

Hardware - Generic Framework

While techically we could use Genode as a thin virtualization layer and just forward all IO to a virtualized traditional kernel (linux) like Xen does, that would not solve the reliability, security and management issues. Porting device drivers to Genode instead of running them in paravirtualized kernels gives certain advantages:

• Isolation - each driver is running in a separate virtual memory space
• Guest (virtualized) kernel can be updated independently of hardware drivers

Therefore, let us take a look at what we need in to have implemented in order to use

I2C

We have currently implemented a basic I2C interface, allowing single- and multi-byte transfers. Among the remaining issues are adding advanced protocols, such as SMBUS and merging the code with Genode upstream.

GPIO Multiplexing

In modern SoCs, GPIO pins are typically multiplexed to several hardware subsystems. Besides, a pin can typically be configured to be pulled either to ground, to voltage source, left floating or in a high-impedance state (tristated). However, the number of available MUX directions, and configuration states differs on various SoCs. For example, TI OMAP has 8 MUX alternative functions and separate configurations for input, output and suspend states, while Qualcomm MSM has only 3 alternatives and one configuration for all states. Currently, we have an implementation for OMAP4, and whether the interface should be made part of GPIO session or kept separate and SoC-specific is under discussion.

IRQ Multiplexing

For some tasks, it is required to have a service that would either combine multiple interrupt sources into a single virtual source or, vice versa, provide multiple virtual interrupts from a single hardware signal. That would be useful for example for I2C port expanders. This could be based on current implementations in Linux.

Voltage Regulators

A framework similiar to the one used in linux should be implemented, including reference counting and disabling the unused power sources.

Framebuffer / Graphics

Currently, Genode supports only a basic fixed framebuffer configuration. Here is a list of features that would be nice to have in no particular order.

• Tiling WM with resizable windows
• Overlay support. If we divert each app to a separate overlay, that would improve performance by using HW blitting capabilities
• HW-accelerated blitting, scaling and other transformations
• Different color spaces support

If this is all implemented, it would be possible to make a generic driver for L4Linux guests that would allow the Guest OS to utilize all hardware capabilities while actual implementation details are hidden in the particular Genode driver.

Another hot topic is OpenGL isolation. The problem with embedded hardware is that the drivers almost often come in the binary form, and running them atop Genode would require writing a custom ELF loader and carefully porting linux kernel-mode drivers. As an experiment, it could be possible to make a small linux ramdisk containing the minimal set of libraries to run a proprietary driver. Also, the linux kernel driver could be used to provide the framebuffer interface to the Genode services. This "linux-as-a-driver" would provide the generic OpenGL interface that could be utilized by other L4Linux instances and Genode services. In general, I think building small instances of linux with the bare minimum of drivers and userspace libraries is a viable alternative to reverse-engineering and porting complicated drivers.

MMC

Not much to say here. Currently there is only one implementation, for OMAP4 ported from u-boot, and it currently only works on pandaboard. It only supports one mmc controller out of five, and only block access, so no fun with wifi. Probably the SDIO stack should be ported from linux as it is quite mature and has several nice properties:

• Supports block, sdio interfaces
• Has quircks and workarounds for broken controllers
• Power Management support
• Cleanly separated bus code from SoC-specific implementation

USB

Currently the only way to use USB is through the Linux DDE (Device Driver Environment) package. This has multiple problems. One is that all the drivers related to usb are linked into a huge blob providing all the interfaces as NIC, Input etc. This clearly violates the microkernel server separation principle.

Another problem is that developing quickly turns into writing quirky hacks to make linux drivers run. USB is quite a complex protocol and porting all drivers to Genode is a complicated task. Therefore, I suggest that for both Host and Client (OTG) modes, only hardware drivers that are able to set up endpoints are written, and then this endpoints are passed to a generic driver inside the virtualized guest OS (linux/BSD) and the guest USB stack takes care of everything. This is not a solution for the long-term goal of using Genode as a general-purpose OS, but would quite suffice for virtualization purposes.

Sound

Sound is a complex issue. I don't currently have a clear vision on this point, here are just some ideas.

• Port alsa-driver to enable porting codec drivers from linux
• Provide a mixer interface and export controls to virtual linuxes
• Alternatively, run PulseAudio on Genode

RTC

This should be pretty straightforward. A driver should be able to:

• Query current date/time
• Store date/time
• Provide Guest L4Linux with date/time via a virtual driver

PM: DVFS

Power Management is a sad topic. There's currently no PM at all which is unsuitable for embedded applications. First step would be implementing voltage and frequency scaling to reduce power consumption when the CPU load is low. For this, the scheduler should send some notifications about the CPU utilization to the policy driver which will in turn decide what to do. Maybe cpufreq from linux should be used for inspiration.

PM: IDLE

For some hardware, especially embedded ARM, it is possible to save tremendous amounts of energy by entering low power states (like WFI - Wait For Interrupt) or CPU-specific ones (like LPA on Samsung Exynos). This task includes adding the support for "mild" power saving states in which caches and devices are not turned off, so we don't have to deal with saving device power state and restoring it on wakeup.

PM: Suspend

Adding the support for deep sleep modes (like S3 suspend to RAM on X86) when the CPU core is turned off is the most complex task. It would involve adding a PM_Session interface for devices supporting runtime PM or needing special actions to save/restore power context. We would also need a policy driver which will decide what to do when some service times out or returns an error while trying to switch a power state.

Input Subsystem

Input session should export more information about the input source type and the L4Linux client driver should set up the input and differentiate between a relative mouse and absolute touchscreen movements dynamically at runtime as opposed to a compile-time ifdef. Some other stuff to add:

• Multitouch support
• GPIO Keyboard driver

LEDs

Should be easy to make a generic LED session driver and L4Linux client for it. I want that the L4Linux driver has no hardcoded defines in it an looks up the available LEDs dynamically. The only thing that would be hard to support but irrelevant are the programmable controllers which allow to set up custom timing patterns. While this may be useful in some cases and save CPU power, these controllers vary in the capabilities provided and usually lack proper documentation. Therefore, I suggest that in case the support for such hardware is desired, it is handled individually for each driver in future.

Sensors

Modern devices come with a variety of sensors: temperature, orientation, acceleration, light level, proximity. Since they are all essentially a stream of data, I suggest a unified interface for them all. The client in L4Linux or other guest OS should look up the sensor parameters and set itself up dynamically. Here is what data should probably be provided:

• Sensor Type : string
• Sensor Data Type : string/enum

Hardware - OMAP4 SoC / Galaxy Nexus phone

To get basic hardware functioning, there remain several "blocker" issues.
Actually, OMAP4 SoC is very complex, so I cannot resist the urge to just map the IO memory and port the linux drivers to Genode "as-is", without rewriting them in a good C++ style and splitting into tiny modules. It seems easier to first create a huge "SOC" driver, and start splitting it into parts when it becomes clear which interfaces and functions are required by other services.

TWL6030 PMIC

Clocks

Power Management

Power Management needs to be implemented not only to conserve power, but also to give access to certain peripherals that are disabled on reboot.

• Voltage Domains
• Voltage Channels
• Power Domains
• PRCM (Power Reset & Clock Module)

Framebuffer

• DSS
• Overlay
• VCS (simplified I2C over DSS) Interface for panel init
• S6E8AA0 Panel

MUSB OTG

Currently, all the platform code for initialization and I2C interface is implemented for dde_linux and the linux musb driver initializes the controller and the device gets recognized by the host. There are several issues to fix still remaining before the driver is functional.

• Timing issues with debugging disabled
• Data endpoints are set up incorrectly, only configuration endpoints work
• RNDIS, Serial and EEM gadget drivers need porting (some functions need to be implemented in dde_linux)

HSI Serial

This is the High-Speed transport to the modem. It is similiar to USB and UART in some aspects.

BCM4330 WiFi

This one depends on TWL6030 and MMC

L4Linux Issues

Speaking of our particular case, running L4Linux atop Fiasco.OC kernel and Genode Framework, I would like to mention the problems which make development a very complicated procedure.
L4Re and L4Linux are developed behind closed doors, and only major releases are dropped to the repository. While linux kernel is typically hosted in a git repository, L4Linux uses SVN. These two factors lead to the loss of commit history. You can't see single changes and bugfixes, only one huge diff. This makes updating to a newer kernel tree manually or rebasing nearly impossible.
Another problem is Genode's ports-foc directory. It contains patches that are applied on top of the 'contrib' directory which is just a clone of L4Linux SVN. Working with separate patches quickly becomes unconvenient.
Here's what I propose and will probably implement for our project.

• Separate L4 architecture patches so that they could be applied on top of any linux kernel version
• Clean up the unused L4 suppor code (which is irrelevant for Genode-based setup)
• Apply the patches on top of vanilla or Android kernel to keep revision history
• Put up a separate kernel repository for L4Linux on Genode

Of course, I would be glad if L4Linux switched to git and made a proper revision history, but convincing the Fiasco.OC crowd is another task, which will probably take a bit more time.

Read More

[programming quickstart]: on programming languages

Introduction
This one is a short essay about my opinion on some of the modern programming languages.
Well, I've not finished it, but decided to post it as is and edit later

With the plethora of programming languages out there many of us are confused about which one to use for their project or as a default language to stick to. Those of us who only start their journey into the fascinating world of programming often find themselves confronted with the question of chosing their first language and I'm sometimes asked to give little advice.

Personally I enjoy trying out every other language I hear about - because I find it fascinating to imagine of what their authors were thinking and how they came to the design. Besides, what I generally like in engineering and IT is that learning a new thing introduces you to the history of the evolution of the industry and makes you get to know and meet a lot of amazing enthusiasts. So I figured I'd go ahead and write a small blog post about various languages I had experience with and my general opinion on the matter.

TLDR:

As usual, the post has some links for further reading at the end (so you can scroll down if you are bored). While I do not consider Wikipedia a reliable source of information, it is still quite useful because it allows you to find links to original papers and you can get a broader knowledge of the subject by skimming through "related" links.

Now, let me tell you some improtant points in case you didn't know or are still confused

• Every algorithm or a problem can be solved in any language.
• Some languages allow you to solve some problems in an easier way
• Some languages bring more fun (positive one as opposed to misery of bug-hunting) to the process of writing code
• It is usually possible to reuse code written in other languages, but the difficulty of the process may vary

What does really matter when choosing a language?

• Popularity. A popular language has a community where you can ask questions.
• Commercial support or maturity. A mature language or the one backed by an enterprise funding is likely to survive for a long time and there's a chance you'll be able to use your code unmodified many years after it's written.
• Available libraries. Ok, in most cases you write algorithms yourself, but sometimes you'd rather shove an existing piece of code rather then reinvent the wheel. It's important that basic libraries for your tasks (UI, Net, data parsing (XML/JSON), media, etc) exist so you can concentrate on solving your problem. 
• Native code integration. Well, it's the key point when choosing anything in engineering - how well it integrates into other commonly used solutions. If your language has a FFI (Foreign Function Interface) which allows you to easily reuse binary libraries written in C/Assembly or there exists an automated code generator allowing to plug in the code written in other languages - chances are you can use it for most of your projects. It is important to notice that some languages/platforms (C#/.Net, ruby/python, Haskell) allow you to easily import native code/data and specify data types (sizes, argument types) directly in code, without having to write a stub C library that would touch the dirty inners of the compiler or a VM (Virtual Machine) like it is done for Java and OCaml.

One important aspect to consider is the type system - that is, how data types and variables are declared and how the compiler/interpreter checks the code correctness. We can classify languages based on the typing system used in several ways:

• dynamic (like ruby which allows to do a lot of stuff at runtime via introspection/reflection [getting type information like available functions and arguments dynamically] at the price of getting a huge crash if you make a typo)
• static (haskell - where all types are checked during compilation and there even exists an opinion that if a program written in a statically typed language compiles, it is correct)
• weak(implicit type conversion like in JavaScript and Perl meaning you can add integers and strings and get integers... weird)
• strict (not even explicit type conversion is possible - like in Haskell/ML which allows to prevent the abuse of type checking)
• explicit (like C, C++, Java) - you have to define the variable types all the time (like, "int x = 1" or "float Sqrt(int x)..." )
• implicit (most functional programming languages like Haskell, F# and recently C# and C++0x) - variable type is deduced from the context [TypeInference]. That is, the compiler tries to find at least some variables for which it can tell the type for sure. Like, integer or float constants. Then it goes back substituting each occurence of the untyped variable with the type of the constant that can be used there. Sort of how Hindley-Milner type inference works [HindleyMilner]

Now, let's go over some languages and discuss each of them individually

Assembly
While you may not encounter the case when you really need to use the assembly language, learning a couple different assemblies (a RISC one like ARM or MIPS and a CISC one like X86) and some specific devices (like, DSP (digital signal processors) or a SIMD (single instruction multiple data which are essentially algebraic operations over vectors of data)) will give you a good insight into how computers work.

Modern compilers are typically good at generating optimized assembly code, so you should really only use assembly where it is needed (like, modifying coprocessor registers, flushing caches) and wrap CPU-specific assembly in the C code so that the major part of your application remains portable.

Please do not fall for those who shout that "assembly is so darn fast I gotta rewrite everything in it". In most cases the performance benefit will be outweight by the complexity of rewriting the code, maintaining it and porting it to a new architecture in future.

two rules of thumb in software optimization:

• Always profile the software (that is, analyze which part of computation, which routine takes most time) before trying to optimize. Remember that Knuth quote, "premature optimization is the root of all evil". That means that you should be aware of the danger of wasting too much time optimizing the wrong part of the program which actually has a negligible influence on performance.
• If some optimization gives you a linear increase in performance, forget about it. You can get the same optimization by doing nothing and just waiting for a next generation of CPUs to come out in half a year. If you really, really need performance, start with trying to decrease algorithmic complexity

C
Now, want or not, you need to know this language and its standard library for it has influenced the design of many subsequent languages and the standard library calls (like fopen, fprintf, fread) are found in the majority of languages (php, ruby, MatLab).

Besides, C is known as 'portable assembly' which means it gives you the precise control over data structures layout in memory, but adds type safety to assembly and you can just recompile your code for any architecture.

In essence, C is the glue that holds together the vast majority of all the languages out there. It is the lingua franca of modern programming.

C++
C++ brings many nice features to C: inheritance, virtual methods, namespaces. It also has a good STL library which has algorithms and data structures which makes it suitable for complex tasks from implementing communication protocols to building large-scale complex frameworks.

Advantages over C:

• more strict type checking
• metaprogramming (template classes)
• STL
• Constructors/Destructors (RAII - resource acquisition is initialization pattern). Kind of makes it easier to clean up memory/file descriptors without the bunch of goto-based error handlers.

However, there are still reasons why C++ may not be the best choice for some projects

• No automatic garbage collector (which may be good for some realtime systems but is generally a shame)
• No stable ABI (application binary interface) meaning binaries compiled with one C++ compiler may not work with the ones made with the another one
• Complex and unclear specification and standard. As of 2012, there's no single compiler conforming fully to any standard revision, and some features like external templates, are not implemented by the major vendors

I would like to point out that there's an amazing Qt4 (now Qt5) framework which has libraries for everything including graphics, media, database access with a semi-automatic memory management which means you can build complex applications as easily as in Java or C# and get all the advantages of the native code (portability, performance, integration)

Java/JVM
Java is one of the many languages implemented on top of a VM (Virtual Machine, a software abstraction that behaves like a real computer), namely JVM.

It has the following nice properties

• Lots of tutorials/quickstarters for beginners
• Huge community of developers
• Backwards compatibility - you can be sure your code still works a couple years after it's written
• Amazing library/dependency management system - Maven
• Comprehensive standard library (runtime) with lots of algorithms
• Good threading support, with the whole range of synchronization primitives and even advanced stuff like memory barriers
• Good support for asynchronous IO (java.nio) for files and sockets
• Own packaging system with package hierarchy and signing. While most *NIX developers find this irritating, there's one huge advantage - you can pack your whole project with all dependencies and configs in a huge jar file and not worry about system updates breaking your application :)
• Fast VM (JVM). Okay, maybe not always as fast as .Net, but more portable (well, at least it runs on linux and OSX), and by orders of magnitude faster than ruby/python. Hence the reason a lot of  developers who enjoy ruby switch to jruby - an implementation of ruby running on top of JVM (hmm.. sounds a bit like off-topic)

However, it is no silver bullet and there may be some reasons why you should avoid Java for some projects

• No good UI toolkit out of the box. Unfortunately JavaFX is not yet (and probably never will be) the part of the standard JRE, and, unlike QML in Qt4 or WPF (Windows Presentation Foundation) in .Net, there's no easy way to create animations, gradients and generally desing the interface in a declarative way (that is, using only a markup language like html/xml without writing code). To some extent NetBeans and Eclipse compensate for that by having intuitive GUI builder tools.
• No optimization for SIMD/vectorized computations (e.g., SSE, NEON) in JVM. Which means the capabilities of modern CPUs allowing for high-speed multimedia processing and power saving cannot be fully utilized until you write in C with JNI
• No builtin syntax for raw memory access (like, unsafe in C#). Which makes interacting with JNI or manipulating raw data (like textures in games) quite complicated
• Strange design solutions violating the so-highly-praised OOP (empty interfaces like Cloneable, erased types for generics in bytecode, mutable final variables -> System.out and friends). While these are not major problems, you should watch out while programming in Java and read documentation very carefully

Overall, Java is a good choice for most tasks, especially multi-threaded web servers due to the rich support for parallel programming, async IO and popular protocols and standards (including HTTP, URL encoding etc). However, it is probably not the best choice for interactive multimedia applications requiring low latency (for example, a music synthesizer) or small destkop apps (because while JVM is very fast, it takes some time to load, and sometimes it does matter whether the app takes 20ms or 2seconds to launch, the users are impatient).

However, the biggest problem is that popular frameworks like Hibernate or Spring are too difficult for beginners because most tutorials don't cover some important issues, and documentation is unfortunately rather vague. It pays off learning all this stuff though - you can set up a complex website  writing virtually no code, relying on the configuration files for customisation.


C#/.Net
C# is often called the "better Java" and .Net in general borrows hugely from the Java world and the JVM architecture. Let's see which strong and weak points are here.

Pros:

• As C# is intended to be the major programming language of the windows platform, .Net CLR (common language runtime) and .Net VM were specially optimized for native code interoperability and multimedia capabilities. Thus, .Net includes the P/Invoke mechanism (DLLImport, typed imports of native functions so that you can use your native code without writing helper stubs in C).
• The language incorporates nice features which make the code more compact and easier to write and read (type inference with the "var" keyword, lambda expressions)
• LINQ. This is a built-in query language which makes interactions with databases and XML very easy. Take a look at the example which shows how easy it is to obtain data from  

 using (NorthwindDataContext context = new NorthwindDataContext())
  {
    var customers =
      from c in context.Customers
      where (c.ContactName.Contains("John")
      &&
      c.CompanyName.Contains("Enterprise")
      select c;
  }

Cons:

• Major updates break binary compatibility. Probably not a huge problem except that you'll have to keep a copy of the older runtime around for legacy apps. I guess the advantages of this decision outweigh the problems
• The default implementation is non-portable, non-crossplatform and not FOSS (free and open source software). Which means you're essentially locked to the Windows ecosystem and depend on Microsoft's design decisions. A typical vendor lock. Something you cannot afford when you need to guarantee your software reliability and availability
• No free (or any) development tools for non-m$ platforms. Even the versions of mono (the FOSS .Net implementation) for Android and iOS cost money, and due to licensing and technical issues such apps will never find their way into application markets.

smalltalk

Smalltalk is an object-oriented language with the long history. Maybe you should try it out just to enrich your knowledge of various design solutions and break your brain

The good

• Nice message-passing model instead of direct method calls. This eventually simplifies null-pointer handling and eliminates the need for type checking in most cases when we know the desired method (or, rather, signal handler) is implemented
• Mixin-based object inheritance. Which means any object can be extended with custom methods, i.e., without inheriting from a superclass
• Many implementations with low footprint. Can even run on bare metal hardware. There exists an operating system written in Smalltalk
• Lots of libraries and bindings to popular libraries

The bad

• Almost extinct today - no developers, no vacancies, no active communities

Perl

I don't like perl. But I'll find a couple minutes to write about it later

PHP
PHP is fairly called the fractal of bad design. And it's not surprising. There are positive moments, of course:

• A lot of ready-to-use libraries, frameworks and web site engines
• A lot of cheap web hostings
• Lots of jobs and vacancies

But they are hugely overweighed by the following problems:

• Weak typing with implicit coercions. This leads to numerous runtime errors
• Eval() support for evaluating a string of PHP code, ability to concatenate PHP code with user-input variables - the result is a multitude of security holes
• Awful standard library, with completely unlogical function namespaces and arguments

Ok, I wanted to write about some other languages, but just leaving the placeholder for now

javascript - pure evil

python - not a bad one, but I've not used it much

ruby - i absolutely love it

Lisp - too much parentheses

Erlang - it rocks

Ocaml/F# - also cool. my tool of choice

Haskell - cool, but too complicated

Now, I'll just reming you to take a brief look at some mathematical software. Might be useful or funny.

Maxima
A nice CAS (Computer Algebra System) written in LISP. Among other cool features it supports symbolic evaluation of expressions and symbolic integration/differentiation. Take a look at the examples and keep getting amazed

Matlab/Octave
Matlab is a commercial general-purpose CAS. Octave is a FOSS implementation of the language which aims to be as compatible with matlab as possible and allow directly reusing matlab code.

Unlike Maxima, Matlab is falls to numeric methods. It has a lot of common algorithms like FFT (Fast Fourier Transform), audio and image processing routines which is why it is commonly used for prototyping algoritms at universities.

One interesting property of Matlab is that internally all data structures are based on vector arrays which closely resembles the SIMD computation unit. Practically this means that parallel operations on multiple data, like adding two vectors, are instantaneous while by-element access is incredibly slow. This makes writing high-performance code in Matlab a challenging task, but it gives valuable experience in optimizing the software for SIMD processors and GPUs so you may want to spend more time practicing it.

Julia

Julia is a very new language similiar to Octave. There are however several reasons why it is worth looking at and has chances to be actually useful and not the lab toy

• It uses LLVM (Low Level Virtual Machine, the latest buzzword in compiler construction and optimization. Chances are it has decent performance
• It has builtin keywords for parallel computation blocks. Like OpenMP for C
• It has the support for distributed computations

R
R is a well known statistical toolbox. It has a huge archive of statistical algorithms at CRAN (Comprehensive R Archive Network, just like CPAN for Perl). It gives you virtually unlimited possibilities to analyze data and draw beautiful plots. Just take a look at [RLinearSquares] and see how easy it is to do a least squares regression analysis with R!

Scripting Languages

Now, let's discuss some scripting languages that are not typically used to write large software but can be used to greatly simplify daily routine tasks (like, file renaming)

AWK is a simple language for text processing. It works by matching a text against a regular expression and then performing an operation on it. It can be used to quickly transform a formatted text
For example, assume we have a text file containing some organization salaries during the year: Name,  Month, Money

Bob 1 2000
Alice 1 2000
Jonh 1 1000
Bob 2 2000
Alice 2 2000
John 2 2500

We could use a simple one-liner to calculate the total amount of money Bob has earned
awk '/Bob/{ sum += $3 } END { print sum }'

Sed
Sed (streaming editor) is similiar to AWK and is also a classical UNIX tool. It is mostly used to replace phrases in text files or live streams. For example, here is how you can replace all the occurrences of "foo" with "bar" in the file named "test.txt"
sed -i s/foo/bar/g test.txt

Bash
Bash is one of many UNIX command shells. It has an imperative syntax much similiar to that of C and Ruby, and allows to automate most daily routine tasks without the need of delving into the depths of operating system internals.

A somewhat not very useful example. Assume you have a lot of files like DSC0001.JPG, DSC0002.JPG from your camera after a summer trip to the seaside. You could rename them altogether to make it easier to recognize them in the mess of media on your hard driver.

for i in DSC*; do mv "$i" "`echo "$i" | sed s/DSC/My trip to south/`"; done  

Programmable Circuits

Let us discuss some languages most people have not even dreamed of. There exists such an interesting area of electrical engineering and computer science as computational logic design. That is, designing the CPUs and all other kinds of VHSICs (very high speed integrated circuits which is nowadays is a collective term for any electronics that are too tiny to see with a naked eye).

From the university course of algebra we know that, in essence, all computations, like additions, multiplications and branching can be represented using common boolean operations. Now, these operations (like conjunction, disjunction and negation) can be implemented electronically as standalone units known as gates. Therefore, we can think in terms of gates with inputs and outputs and ignore the electrical characteristics of the circuit (remember, we live in the idealized digital world which is a no-brainer).

So, instead of writing a huge diagram comprising miles of wiring and tons of paper to print it out, we can try to make a language that would compile itself into boolean functions. And turns out, this has been done long ago and that's how modern electronics design is done. This is called HDL which stands for the Hardware Description Language.

There exist two major HDLs: verilog and VHDL (actually, there exist some others like SystemC and SystemVerilog, but they are less widely used and eventually there's little practical difference between all of them). The primary difference between them is that verilog is weakly and implicitly typed and programming in it feels like a mix of C and Erlang and VHDL is strictly typed and gives a feeling of Haskell or Delphi.

I guess we'll leave examples till I write the article on circuit design basics.

Note to self: gotta explain transistors, npn vs pnp vs FET, clocks, latches. maybe write a post about electronics and circuit design?

To sum up,
as most of the stuff I do is either programming microcontrollers, writing OS drivers and playing with DSP and computer graphics (opengl), as I am using linux and prefer simplicity, my languages of choice are the following:

C - for most small tools and drivers
C++ - for complex projects involving UI and when I don't want to reinvent OOP with C-style macros

ruby - for simple text processing and to prototype algorithms

OCaml - for writing parsers and interpreters
Octave - instead of a calculator

References

[HindleyMilner]

[TypeInference] http://en.wikipedia.org/wiki/Type_inference

[TuringTarpit] http://c2.com/cgi/wiki?TuringTarpit

[WAT] https://www.destroyallsoftware.com/talks/wat

[Dart] http://www.dartlang.org/
[LiveScript] http://livescript.net/
[SqueqkNOS] http://squeaknos.blogspot.ru/

[Sedtris] http://uuner.doslash.org/forfun/sedtris.sed

[PerlUndecidable] http://www.perlmonks.org/?node_id=663393
[RLinearSquares] http://www.cyclismo.org/tutorial/R/linearLeastSquares.html

Read More

audio update

So I got fed up by both my smartphone's audio quality and my headphones and decided I need a decent player

I got a Cowon iAUDIO 9 and Koss Porta Pro headphones. Really, I wanted to get iAUDIO 10, but there was none in the shopping centre and I wanted to get it today and enjoy the music ASAP.
http://www.cowonglobal.com/product_wide/iAUDIO9/product_page_1.php

As for the headphones, I find Porta Pro quite nice. For $30 they're the best. Their frequency response shows no ringing in the high frequency range and they have very powerful low frequencies (which some people find rather a disadvantage). The drawback is that for such clear sound the price is the 60Ohm impedance making them sound quiet on most players. I was having AKG K430 for some months and find them not so pleasant to use. There are almost no low frequencies and the sound in high frequency range is a bit distorted. Besides, Porta Pros look cool and stylish. But they're made of a very weak plastic and break easily. If it were not for their low price, I would never buy them or recommend to anyone, but as you see, I've fallen for them for the second time.

Some minor problems I've found:

1. USB access is slow. no, it's SLOW. took the whole hour to copy the 9GB of music
2. Player menu is a bit slow. The screen lags behind the actual input handling. If you click "play" and then scroll through the playlist, it will actually rewind the music track
3. Not a problem, but a counterintuitive feature. Rewinding at the start of track does not switch to the start of the previous track, but rewinds to some last seconds of it.

The good points however are:

1. Output power and SNR are above average (30mW and 95dB for reference). Enough to driver even the 60Ohm Porta Pros.
2. Easy to use menu controlled with two keys and the scrolling area
3. Supports directory browsing (as opposed to album/library browsing which is the only available option on most players today)
4. Easy to control the volume and tracks with hardware keys while it is in the pocket
5. supports FLAC

Overall, I'm very happy and find the sound quality good - it is loud and frequency response is more or less flat, so the sounds of all frequency range are audible, there's no lack of bass and the sound is detailed because high frequencies are not exceedingly powerful and middle ones are not silenced like it often happens.

So, why was I dissatisfied with my Galaxy S2?

1. output power is low and frequency response is far from flat. This means sound is too quiet in the headphones and one needs to tweak the equalizer to make low frequencies ("bass") audible and high frequencies not such irritating
2. it is running Android. I dunno why, but after I've installed a couple softwares (including Twitter and  Foursquare) and enabled GMail synchronization, my phone is periodically becoming extremely slow. So sometimes even the music playback is quirky
3. it is running Android. when I copy the music via USB using it in Mass Storage mode (USB Flash Disk emulation mode), I need to reboot the phone a couple times or force the media refresh via the developer menu to make the player see the files
4. it has a dual-core CPU. and a huge screen. and yes, it is running Android. When you enable 3G, it does not even last the whole day. And takes too much space. Switching tracks on the go is inconvenient.

What I did not like about my previous phone (Sony Ericsson Xperia X1)?

1. Output power is too low to drive Porta Pros
2. Quirky sound. Actually, the problem is that the sound system of the phone uses multiple buffering and a complex synchronization scheme for them. The fact is that there are two processors, the ARM11 core running Windows Mobile (or linux/Android thanks to our work at htc-linux.org) and the ARM9 running an L4 kernel which manages some peripherals including audio. Audio is controlled via RPC. Sometimes, and unfortunately too often for me to ignore, a buffer underrun happens. That is, when the audio system is expecting N samples in the queue, but only M are available, and M is less than N. When that happens, you hear a clicking noise or some random stuff that was left in the buffer before. Unfortunately on most devices having MSM7200A CPU this issue occurs approximately once in some 10-15 minutes making sound awful.

Read More

Results of 2012 as seen from 2013

This post is a couple hours late but my excuse
is that I was having amazing holidays with my nearest and dearest
and without the huge distraction of the Internet.

Now that another year is gone, it's time to ask ourselves of what
memorable things are left after it, what we're leaving behind, what
we can be proud of and whom to thank for what.

For me, 2012 was full of miracles and excitement. If I were asked
to come up with a list of things I want to thank the past year for, here
are the things I'm happy about having done:

Got rid of stupid fears and prejudice.
Sometimes you just gotta sit down
and say to yourself: "C'mon, it's no fun,
stop moaning and start doing what you want."
This alone let me make my dreams come true.

Became myself again. Doing what's fun and useful
and not getting drowned by what's boring and stupid.

Completed several long-term hobby projects and feel
extremely proud of the results. Especially of the indirect
results, i.e., finding new friends and acquaintances in process.

Met a lot of interesting people with whom I feel comfortable to discuss
any ideas and who more importantly can give support (most important kind
of support is moral support of course, but it's not limited to it) for
starting any wildest project.

Got the job of my dream. While some may regard talking about or
focusing on job as utterly boring, I just find it marvelous
that I'm getting a chance to talk to a lot of enthusiasts,
learn from them and be able to get in touch with any kind
of stuff I've always dreamt of. And of course it's quite
pleasant to get paid some money for doing what I like.

Figured out electronics and physics in general are
actually easy and fascinating. Gonna spend more
time studying them and finding ways to apply
this knowledge to bring more joy and fun
to everyday life.

Started making first steps into the world of music.
Feels like the beginning of a new wonderful
journey and the eternal love.

Found my happiness in someone who inspires me.
It never ceases to amaze me how little
we know the ones we're spending
much time together.

Read More

thoughts on modern operating systems

For a long time I've been messing with low-level code on various embedded hardware.
While I generally find linux kernel code an outstanding example of software architecture
and one of the cleanest projects implemented in the C language, there is
still room for improvement.

I'll try to summarize my impressions on developing for linux kernel (and other bare-metal
software including u-boot, lk, netbsd (the one of two BSDs which are not completely irrelevant ancient shit, btw. Second one is DragonflyBSD)). In this post I want to consider two aspects
of an operating system kernel implementation: architectural issued (which are language-
agnostic) and the choice of a programming language.

One point of argument is how modular the kernel should be. The two major models are the monolithic kernel (where all drivers are executing in the same virtual memory space and consequently there is no isolation between the access to resources between drivers) and the microkernel approach where
every driver is running as a process in a separate address space in user-mode.

The problems of the monolithic kernels

• All drivers have the same access to all resources. Which means, if we don't trust the driver code (say, a closed-source driver), our entire system is compromised. Moreover, any error in the driver code is a potential root exploit because if we manage to smash stack or force the kernel code to jump to a usermode helper, we have all priviliges possible [KernelExploits].
• If a driver code hangs, we cannot easily restart it because all the code is running inside the single process and shares the global variables. That is, even if we remove and insert the module, it does not guarantee that we start over with the clean state. Moreover, shared locks will in most cases prohibit us from doing it when things go bad.

On the other hand, microkernels try to solve the problem by the following security measures:

• Running each process in the user-mode to prevent it peeking into other processes data by exploiting supervisor mode
• The principle of least priviliges: only grant the access to the resources needed by the driver (IO, GPIO, IRQ)
• Reducing TCB (trusted computing base) to the minimum amount of code
• Using capabilities to control access to resources. Capabilities are, like, pointers to the desired resource, with the ownership controlled by the kernel (or, actually, by the policy manager which can comprise multiple entities inside and outside of kernel)

There exist various microkernels. Most commercially successful are the, of course, the QNX and Mach kernels. While QNX and Mach can be used in a 'pure' microkernel sense - that is, when nothing runs in kernel in suprevisor mode and has ultimate priviliges, most practical applications include interrupt routines and drivers in kernel (QNX allows it, and Max OS X kernel, XNU, relies heavily on this feature).

One of the most interesting projects are the L4 Fiasco.OC [FOC] kernel from TU Dresden and the Genode Framework from Genode labs.

Fiasco.OC is not particularly interesting per se (there are far more interesting projects like l4.verified from NICTA which is formally prooven), but it has quite a set of nice properties

• It is opensource. Well, kind of. The development is done behind closed doors and it's a bit of pain to push a patch to upstream but at least we can download the source code and adapt it for our purposes
• It has L4Re (L4 Runtime Environment) which is a set of libraries to ease porting software from conventional POSIX-compatible environments
• It can be used to paravirtualize other kernels. Most notably, there exists the [L4Linux] project allowing one to run linux on top of l4 to allow seamless code reuse.

Genode Framework is a step aside from a multitude of attempts to write a successful microkernel capable of surviving in the cruel world. It can run on top of many kernels (L4 from TU Dresden, OKL4, linux) and even directly on bare-metal hardware and provides a unified capability-based security model on top of any kernel it uses. It has some nice features.

• Supports ARM architecture and recent SoC (OMAP4). This means we can use it for paravirtualizing, say, Android on armv7 CPUs with no hardware virtualization support. That's important because ARM does not have such a choice of virtualization solutions like, say, x86.
• Implemented in C++ using templates. Enforcing type checks at compile time reduces the amount of programming errors
• Has a nice RPC/IDL implementation in C++. Programming multi-server software has never been that easy.

Unfortunately, there's a heckuva lot of work ahead if we want to use Genode for world domination. Here are some ideas I have about improving the current state of Genode framework and I do hope I'll get a chance to work on some of these things eventually. Most of the troubles are caused by the poor hardware support.

• Lack of device drivers. What should be done is to write abstractions (interfaces and sample implementations) for various hardware classes - voltage regulators, MMC cards, GPIO controllers et cetera. Like in linux, but using kernelization (decomposing software into standalone servers communicating via a trusted RPC guarded by security tokens - capabilities) for improving security and C++ or a custom DSL for enforcing type checks and preventing various programming errors. Ideally, I would love that we could enforce coarse-grained access policies for every resource - be that a GPIO pin or an I2C device register.
• No power management. I mean, at all. Not even DVFS (Dynamic Voltage/Frequency Scaling). No suspend-to-ram support. Which quite limits the use of the framework on mobile devices.
• Poor display management support. I propose that nitpicker or some other display server should be extended to allow hotplug, runtime resolution switching and l4linux/genode framebuffer driver should be completed to allow the use of all the complex features of modern display adapters - blitting, rotation, overlays and what not.
• Proper fault handling. I think some kind of a system (call it a paradigm, a design pattern or whatever) should be implemented to gracefully restart services in the case of failure (it is indeed very difficult to decide what to do when a driver fails). There should probably be some abstract policy class with various implementation. One implementation (for example, for the user desktop) would try to restart failing tasks. The other one (say, for trusted computing environments) would kill the world if any single process fails.
• Some nice UI should be written for easily switching and creating new virtualized linux instances.

One huge security hole worth mentioning is hardware. Not only it's very hard to verify the lack of backdoors in hardware (a modern smartphone actually has more than 20 CPUs of various architectures running closed-source code) but some hardware features can be used to circumvent software security measures.

The most famous kind of hardware-assisted attacks are the DMA-based attacks. In fact, in older systems, a DMA controller typically had an unrestricted access to all physical memory making it possible for a malicious driver to control the whole system. Moreover, some protocols, like ieee1394 (FireWire) and PCI by design allowed arbitrary device to access all system memory. There have been various exploits to circumvent desktop operating systems protection (like, [WinLockPwn]). Moreover, the same techniques have been successfully used to crack the security and hypervisors of various gaming consoles as shown in [Console] and [StateOfTheHack]. One solution to this is using IOMMU which is a virtual memory translation buffer between the device and the IO bus. Besides the ability to define protection domains on a per-page basis it also allows using discontiguous memory regions with the devices that don't support it (like, most of camera ISPs (image signal processor) or GPUs) and remapping hardware memory into arbitrary process (as an example, this allows you to pass through your VGA card into a paravirtualized instance of linux on Xen)

The other flame-provoking area is the choice of a programming language for implementing
the operating system. Historically most kernels, including Linux, *BSD and to some extent
Windows NT, were written in plain C, while most of the others.

To be honest, I'm sick to death of programming in the C language. While C is often called the 'cross-platform assembly' and you can always understand what your code will get translated into, it offers a multitude of ways to shot yourself in feet and I have some ideas about what a language for kernel-level programming should look like. I'll divide the ideas into two sides: the ones will focus on improving the language in a non-intrusive way, without breaking semantics and code efficiency. The others will discuss the alternative approaches to system software construction.

First off, let's discuss common problems of programming in C and ideas on improvement. Here's what's been bothering me daily since I started my journey into the world of embedded development.

• Uninitialized memory. Seriously, this is the cause of 90% of misbehaving software. Just enable the corresponding flag if your compiler supports it. I wish this were always enforced by the C compiler. The performance penalty is negligible im most cases. When one needs uninitialized data or it is impossible to store the initial data values in the BSS or clear the buffer at runtime, a separate language construct should be used so that it is possible to identify all such places at compile time.
• Buffer Overflows. This is one huge security issue. In most cases it is caused by the errors in handling null-terminated strings (whoever thinks they're a good idea is probably a cruel genius hoping to have a backdoor everywhere). One possible way to work around would be to use typed wrappers for arrays - a data structure containing array pointer and element count with accessors and mutators checking for out-of-range errors. Like in Java, but this can be perfectly well simulated using C++ templates or less comfortably with C macros.
• Null pointer dereferences. All pointers should be checked for validity by the compiler. Besides, casting pointers to integers should be banned, and IO buffers should also be implemented within the compiler so that a programmer does not mess manually with memory.
• Lack of both OOP classes, Haskell type classes and functional-style pattern matching. While combining all of these features usually makes the language very complicated (like Ocaml), the lack of any of them makes programming a constant exercise in reinventing the compiler with the help of fugly (or beautiful) macros like container_of.
• (C++) it would be nice to be able to define abstract class implementations at compile-time without vtbl. To some extent you can get away with using templates for substituting types in class definition. C++ templates in generally is a powerful system similiar to hindley-millner type inference systems in ML or Haskell, use it wisely :) 
• Stupid breaking up code into the header and the 'implementation'. This is especially tiring in C++, because you have to jump between the header and the source for any change. I'm not even going deep into discussing other issues. Like, most C++ compilers being unable to handle external templates.

Now, let's consider alternative approaches, their strong and weak points. Here are some ideas that we could borrow from the userland world.

• Garbage Collection. This alone can save a lot of memory leaks and debugging efforts. While it is true that GC activity is quite hard to predict and there's been a (false) belief that GC and realtime systems are incompatible, it should be noted that manual memory management also suffers from VM thrashing and malloc() calls can consume arbitrary amount of time. Therefore, in some cases where performance and reliability are the question of life and death, dynamic memory allocation is not used at all and all memory buffers are statically defined at compile time.
• Runtime type introspection (reflection). It is very useful to be able to query object state without writing tons of macro boilerplate. While we can put the task of type-checking on the compiler, sometimes we need to interact with remote code and userspace (e.g., sysfs) and then RTTI comes handy. For example, most userlands use some kind of a typed message bus (dbus, p9fs) for IPC (inter-process communication) which we could plug directly into the kernel public interface

• Monitor primitive for atomic/critical sections to avoid manual locking control. In fact, I want something like the 'synchronized' primitive in Java or 'lock' C#. Besides, it would be very helpful if the monitor knew the execution context (interrupt, bottom part handlers) and use the sleeping or non-sleeping locking accordingly (spinlocks vs mutexes). Like: "atomic [x <- readBus(); waitBus(x & ~1)]" 

There exist multiple projects aiming to exploit the advantages of very high-level programming languages to improve the quality of system software. To name just a few most interesting for me:

• .Net Micro framework by Microsoft [DotNetMicro] which runs on bare hardware. It requires just around 64K RAM. While this is a huge waste of computational power for a tiny controller like Atmega 48, it is still smaller than most fully-featured kernels like Linux, Windows NT or hypervisors like Xen or L4. The nice thing about this project is that it is open-source and supports a lot of common hardware (pardon my ignorance, but the last time I checked 5 years ago it supported the Intel/Marvell PXA SoC which was still quite popular back then)
• Singularity OS by Microsoft. In fact, it is similiar to .Net Micro in that it uses the dialect of the C# programming languages, but it also relies on the Ahead-Of-Time (AOT) precompilation to native code.
• House is an operating system implemented in the Haskell programming language [HouseOs]. This is quite interesting because Haskell is a strongly-typed language with type inference which makes it ideal for writing large-scale complex systems. Unfortunately it seems the development rate has slowed down lately and hardware support is limited to a subset of x86-compatible PCs
• [SqueakNOS] is an operating system built around the Squeak Smalltalk environment running on bare metal, with everything down to interrupt handling done in Smalltalk. Smalltalk is a very nice object-oriented language (in spite of the bad fame OOP has gained recently due to the perverted nature of C++ and Java) built around the signal-slot calling mechanism.

Further reading:
[KernelExploits] Writing kernel exploits http://ugcs.net/~keegan/talks/kernel-exploit/talk.pdf

[FOC] TU Dresden Fiasco.OC L4 kernel website http://os.inf.tu-dresden.de/fiasco/overview.html
[L4Linux] TU Dresden L4Linux website http://os.inf.tu-dresden.de/L4/LinuxOnL4/
[Genode] Genode Framework official website http://genode.org/

[WinLockPwn] Adventures with Daisy in Thunderbolt-DMA-land: Hacking Macs through the Thunderbolt interface http://www.breaknenter.org/tag/winlockpwn/
[Console] Modern Game Console Exploitation http://www.cs.arizona.edu/~collberg/Teaching/466-566/2012/Resources/presentations/2012/topic1-final/report.pdf
[StateOfTheHack] All your consoles are belong to us http://delroth.net/state-of-the-hack.pdf

[DotNetMicro] Microsoft .NET Micro Framework http://www.netmf.com/
[Singularity] Microsoft Singularity OS http://research.microsoft.com/en-us/projects/singularity/
[HouseOs] House Operating System website http://programatica.cs.pdx.edu/House/
[SqueakNos] SqueakNos - Smalltalk on bare hardware http://squeaknos.blogspot.com/

Read More

Doing it wrong: application preferences

One thing that's been always bothering me is the lack of a unified way to keep persistent settings.

When you're installing a shiny new piece of software, you never know which surprises it brings. The location and syntax of the configuration files varies between applications and environments. Let's take a look at the most commonly used persistent storage formats and analyze their strong and weak points. Note that I only want to discuss the issue only in the aspect of desktop usage, where usability and maintainability should not be sacrificed over performance as we do in the embedded world. Also, note that I'm mainly focusing on the formats used in the desktop *NIX distributions because that's what I'm mostly using.

One key difference between many formats is whether they allow text data with arbitrary formatting and spacing or have a fixed binary layout. The former is obviously human-readable (and sometimes, editable), the latter can in most cases be easier parsed by the machine.

Plain text
Pro:

• Human-readable

Contra:

• No unified syntax (everyone invents their own)
• No way to contol the correctness of the file contents (as opposed to xml schema for which there exists a multitude of tools and libraries)

XML
+

• Can be edited by a human (to some extent)
• Strict structure definition is a part of a standard (user can define custom types and limits for variable values)
• Supported by many tools

-

• Hard to read by a human
• Inefficient to parse (though, better than a plain text without schema)

GConf is an XML-based configuration system for the GNOME and GTK applications. Basically, it inherits all the pluses and minuses of the XML.

JSON
+

• Easy to read and edit by a human
• Easy API in most libraries (we can think of it either as of a tree of objects or as hashmap of hashmaps of hashmaps...)

-

• Lack of built-in schema checking (i.e., it has a formal syntax and we can check that a JSON is valid, but we cannot verify that the data structures are initialized with correct values)

Now, let's discuss binary formats.

Custom formats

Windows Registry
+

• It's fast.

-

• Most system-critical data is kept inside a single file (well, actually around 6 files in %windir%\System32\config). A single failure means the system is unable to boot.
• All applications store their data inside a single database. If an application is running with 'Administrator' (something like root, but for retards), it can easily modify or read the data of other applications. This is a huge security issue.
• Application activity is not logged (at least, by default). It means mischievous software leaves trails even after it is uninstalled which is especially true of shareware and other crapware.
• Application and system registry are not isolated. You cannot just reinstall the system without keeping all preferences (like, keep the home dir but replace the rootfs in *NIX)
• There are only several basic data types (DWORD, Text, Binary) and the user cannot introduce their own

DConf is a replacement for GConf from the GNOME project. The only real difference is using a binary format instead of XML. While it is arguably faster, it is completely incomprehensible to a human-being and when things go bad, I have no idea what to do.

BSON is a binary version of JSON.
+

• It is faster
• It consumes less space

-

• It introduces custom types (like UUID) which means that any JSON can be converted to BSON, but not the other way round.

MessagePack is a binary serialization format that is aiming to be compatible with BSON while being optimized for network usage
+

• It offers high compression ratio reducing network traffic
• Comes with the IDL/serialization code for most mainstream languages (C++, ruby, python)
• Offers type checking when used with statically typed languages
• Comes with an RPC implementation which can be useful for network applications

-

• Performance can theoretically be a bit lower than BSON due to bit-packing 

There of course exists a multitude of other config formats, but the most used on a linux destkop are plain text and GConf. Therefore, I'm lazy to describe every other thing under the sun.

Now, here is what I think application developers should do

• Use glib/qt4/java built-in APIs for keeping preferences. Don't reinvent the wheel
• Don't break the config format with time (ideally, you don't know anything about the config format if you use the existing API)
• Make it easy for the users to backup the configuration data

Ideally, I would like that all apps (at least, all the free and open-source apps on linux) would use the same API for keeping preferences so that the system administrator can choose various backends, be it xml, json, sql or what not without modifying the applications.

Read More