Bloerg BlogA blog about technology … mostly programmingZola2024-02-25T00:00:00+00:00https://bloerg.net/atom.xmlInspecting streams for hashing purposes2024-02-25T00:00:00+00:002024-02-25T00:00:00+00:00
Unknown
https://bloerg.net/posts/inspecting-streams-for-hashing-purposes/<p>This is a personal note on how to inspect async streams in order to hash the
contents without storing and reading the stream once more.
This might become the
start of a a series of little code snippets that might be interesting to others
as well.</p>
<h3 id="code">Code</h3>
<p>As an example fetch a URL and compute the SHA256 hash on it:</p>
<pre data-lang="rust" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#72ab00;">use </span><span>anyhow::anyhow;
</span><span style="color:#72ab00;">use </span><span>futures::StreamExt;
</span><span style="color:#72ab00;">use </span><span>sha2::Digest;
</span><span style="color:#72ab00;">use </span><span>std::path::PathBuf;
</span><span>
</span><span>#[</span><span style="color:#5597d6;">tokio</span><span>::</span><span style="color:#5597d6;">main</span><span>(flavor </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">"current_thread"</span><span>)]
</span><span>async </span><span style="color:#668f14;">fn </span><span style="color:#c23f31;">main</span><span>() -> </span><span style="color:#a2a001;">Result</span><span><(), anyhow::Error> {
</span><span> </span><span style="color:#668f14;">let</span><span> arg </span><span style="color:#72ab00;">= </span><span>std::env::args()
</span><span> .</span><span style="color:#b39f04;">skip</span><span>(</span><span style="color:#b3933a;">1</span><span>)
</span><span> .</span><span style="color:#b39f04;">next</span><span>()
</span><span> .</span><span style="color:#b39f04;">ok_or</span><span>(</span><span style="color:#a2a001;">anyhow!</span><span>(</span><span style="color:#d07711;">"no URL given"</span><span>))</span><span style="color:#72ab00;">?</span><span>;
</span><span>
</span><span> </span><span style="color:#668f14;">let</span><span> url </span><span style="color:#72ab00;">= </span><span>url::Url::parse(</span><span style="color:#72ab00;">&</span><span>arg)</span><span style="color:#72ab00;">?</span><span>;
</span><span> </span><span style="color:#668f14;">let</span><span> path </span><span style="color:#72ab00;">= </span><span>PathBuf::try_from(url.</span><span style="color:#b39f04;">path</span><span>())</span><span style="color:#72ab00;">?</span><span>;
</span><span> </span><span style="color:#668f14;">let</span><span> filename </span><span style="color:#72ab00;">=</span><span> path
</span><span> .</span><span style="color:#b39f04;">file_name</span><span>()
</span><span> .</span><span style="color:#b39f04;">ok_or</span><span>(</span><span style="color:#a2a001;">anyhow!</span><span>(</span><span style="color:#d07711;">"URL does not contain a filename"</span><span>))</span><span style="color:#72ab00;">?</span><span>;
</span><span> </span><span style="color:#668f14;">let mut</span><span> hasher </span><span style="color:#72ab00;">= </span><span>sha2::Sha256::new();
</span><span>
</span><span> </span><span style="color:#668f14;">let</span><span> stream </span><span style="color:#72ab00;">= </span><span>reqwest::get(url)
</span><span> .await</span><span style="color:#72ab00;">?
</span><span> .</span><span style="color:#b39f04;">bytes_stream</span><span>()
</span><span> .</span><span style="color:#b39f04;">inspect</span><span>(|</span><span style="color:#5597d6;">bytes</span><span>| {
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#668f14;">let </span><span style="color:#a2a001;">Ok</span><span>(bytes) </span><span style="color:#72ab00;">=</span><span> bytes {
</span><span> hasher.</span><span style="color:#b39f04;">update</span><span>(bytes);
</span><span> }
</span><span> })
</span><span> .</span><span style="color:#b39f04;">map</span><span>(|</span><span style="color:#5597d6;">chunk</span><span>| chunk.</span><span style="color:#b39f04;">map_err</span><span>(|</span><span style="color:#5597d6;">err</span><span>| std::io::Error::new(std::io::ErrorKind::Other, err)));
</span><span>
</span><span> </span><span style="color:#668f14;">let mut</span><span> reader </span><span style="color:#72ab00;">= </span><span>tokio_util::io::StreamReader::new(stream);
</span><span> </span><span style="color:#668f14;">let mut</span><span> file </span><span style="color:#72ab00;">= </span><span>tokio::io::BufWriter::new(tokio::fs::File::create(filename).await</span><span style="color:#72ab00;">?</span><span>);
</span><span> tokio::io::copy(</span><span style="color:#72ab00;">&</span><span style="color:#668f14;">mut</span><span> reader, </span><span style="color:#72ab00;">&</span><span style="color:#668f14;">mut</span><span> file).await</span><span style="color:#72ab00;">?</span><span>;
</span><span>
</span><span> </span><span style="color:#668f14;">let</span><span> sum </span><span style="color:#72ab00;">=</span><span> hasher.</span><span style="color:#b39f04;">finalize</span><span>();
</span><span> </span><span style="color:#a2a001;">println!</span><span>(</span><span style="color:#d07711;">"</span><span style="color:#aeb52b;">{} {:?}</span><span style="color:#d07711;">"</span><span>, hex::encode(</span><span style="color:#72ab00;">&</span><span>sum), filename);
</span><span>
</span><span> </span><span style="color:#a2a001;">Ok</span><span>(())
</span><span>}
</span></code></pre>
<h3 id="cargo-toml-dependencies">Cargo.toml dependencies</h3>
<pre data-lang="toml" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-toml "><code class="language-toml" data-lang="toml"><span>[</span><span style="color:#6486ab;">dependencies</span><span>]
</span><span style="color:#7f902a;">anyhow </span><span>= </span><span style="color:#d07711;">"1.0.80"
</span><span style="color:#7f902a;">futures </span><span>= </span><span style="color:#d07711;">"0.3.30"
</span><span style="color:#7f902a;">hex </span><span>= </span><span style="color:#d07711;">"0.4.3"
</span><span style="color:#7f902a;">reqwest </span><span>= { </span><span style="color:#7f902a;">version </span><span>= </span><span style="color:#d07711;">"0.11.24"</span><span>, </span><span style="color:#7f902a;">features </span><span>= [</span><span style="color:#d07711;">"stream"</span><span>] }
</span><span style="color:#7f902a;">sha2 </span><span>= </span><span style="color:#d07711;">"0.10.8"
</span><span style="color:#7f902a;">tokio </span><span>= { </span><span style="color:#7f902a;">version </span><span>= </span><span style="color:#d07711;">"1.36.0"</span><span>, </span><span style="color:#7f902a;">features </span><span>= [</span><span style="color:#d07711;">"macros"</span><span>] }
</span><span style="color:#7f902a;">tokio-util </span><span>= { </span><span style="color:#7f902a;">version </span><span>= </span><span style="color:#d07711;">"0.7.10"</span><span>, </span><span style="color:#7f902a;">features </span><span>= [</span><span style="color:#d07711;">"io"</span><span>] }
</span><span style="color:#7f902a;">url </span><span>= </span><span style="color:#d07711;">"2.5.0"
</span></code></pre>
Building rusticl2022-11-20T00:00:00+00:002022-11-20T00:00:00+00:00
Unknown
https://bloerg.net/posts/building-rusticl/<p>OpenCL has been with me for more than a decade, back when we decided to use it
in our research project to make it the foundation for accelerating synchrotron
imaging. Now, as history has shown, OpenCL never really took off, partially
because Apple (the initial sponsor) dropped it but more importantly NVIDIA being
very successful in locking in people with their proprietary CUDA solution.
Nevertheless, support by all major GPU vendors is there to some degree, so
software <em>can</em> be accelerated in a somewhat portable way. The degree AMD has
taken is somewhat questionable though: they do support OpenCL either via their
open ROCm stack but just for select GPUs and short support windows or via their
proprietary amdgpu-pro packages. The latter is what I use today to enable OpenCL
in Darktable but it is a hack because it involves downloading Debian packages
from their website and extracting them correctly.</p>
<p>Fast forward to 2022, Rust is on its way to become the premier systems language
and heroes like Karol Herbst start writing <a href="https://docs.mesa3d.org/rusticl">OpenCL mesa
drivers</a> completely alleviating the need for
the crap AMD is offering (well almost). Because building and using it is not
very straightforward at the moment, here are some hints how to do that. I am
assuming an older Ubuntu 20.04 box, so some things could be in the 22.04 repos
already.</p>
<h3 id="installing-tools-and-dependencies">Installing tools and dependencies</h3>
<p>Add the LLVM apt repos</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>deb http://apt.llvm.org/focal/ llvm-toolchain-focal-15 main
</span><span>deb-src http://apt.llvm.org/focal/ llvm-toolchain-focal-15 main
</span></code></pre>
<p>to <code>/etc/apt/sources.list</code> and run <code>apt update</code>. Install</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ apt install clang-15 libclang-15-dev llvm-15 llvm-15-dev llvm-15-tools
</span></code></pre>
<p>Ubuntu 20.04 comes with a pretty old version of meson, so lets create a
virtualenv and install it along with mako which is used by mesa itself:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ python3 -mvenv .venv
</span><span>$ source .venv/bin/activate
</span><span>$ pip3 install meson mako
</span></code></pre>
<p>We also need bindgen to bind to C functions but luckily the <code>bindgen</code> program is
sufficient and can be installed easily with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ cargo install bindgen-cli
</span></code></pre>
<h3 id="build-rusticl">Build rusticl</h3>
<p>At the moment the radeonsi changes are not yet merged into the main branch,
hence</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ git remote add superhero https://gitlab.freedesktop.org/karolherbst/mesa.git
</span><span>$ git fetch superhero
</span><span>$ git checkout -t rusticl/si
</span></code></pre>
<p>For some reason, rusticl won’t build with the LLVM 15 libraries as is and we
have to add <code>clangSupport</code> to <code>src/gallium/targets/opencl/meson.build</code>
as yet another clang library to link against in order to find some RISCV
symbols. It’s time to configure the build with meson</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ meson .. -Dgallium-rusticl=true -Dllvm=enabled -Drust_std=2021 -Dvalgrind=disabled
</span></code></pre>
<p>Note that meson does not check for existence for Valgrind on Ubuntu and enables
it by default causing build errors when the development libraries are not
installed. Time to build and install using <code>ninja</code></p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ ninja build && ninja install
</span></code></pre>
<h3 id="running-opencl-programs">Running OpenCL programs</h3>
<p>I tend to install mesa into a custom prefix and pre-load it with my old <a href="/posts/virtual-environments-for-compiled-libraries-and-applications">shell
script</a>. In
order to have the system-wide ICD loader find the ICD that points to rusticl, we
have to set the <code>OPENCL_VENDOR_PATH</code> environment variable to the directory
containing the .icd, i.e. <code><some-prefix>/etc/OpenCL/vendors</code>. Also we have to
set the <code>RUSTICL_ENABLE</code> environment variable to <code>radeonsi</code> because it is not
enabled by default yet. With that set <code>clinfo</code> should show a platform with the
name <code>rusticl</code>.</p>
<h3 id="setting-up-rust-analyzer">Setting up rust-analyzer</h3>
<p>If you intend to dig into rusticl itself you will notice that this is not your
bog standard Cargo project but intertwined with meson which takes care of
building the majority of the C and C++ sources. Because of this rust-analyzer is
not able to figure out the structure of the rusticl project. Luckily, meson 0.64
produces a <code>rust-project.json</code> file that describes the structure but
unfortunately the paths in there seem to be a bit messed up. After symlinking
from the root of the Git repo (so rust-analyzer can find it) and changing the
paths to point to existing directories, rust-analyzer was able to make sense of
the project.</p>
wastebin 2: electric boogaloo2022-08-29T00:00:00+00:002022-08-29T00:00:00+00:00
Unknown
https://bloerg.net/posts/wastebin-2/<p>It has been almost a month already since I released the first major breaking
release of my minimalist <a href="https://github.com/matze/wastebin">pastebin</a>. The main
reason to bump the major version was due to streamlining routes especially
dropping the <code>/api</code> ones and adding query parameters where it made sense. In
between my last post and version two, there have been many other non-breaking
changes like correct caching (of course …), more keybinds, better looking user
interface, minor fixes and a demo site hosted <a href="https://bin.bloerg.net">here</a>.</p>
<p>Currently, I am preparing everything to make the move to the upcoming breaking
0.6 release of axum. But more importantly, I am investigating ideas how to get
rid of <a href="https://github.com/trishume/syntect">syntect</a>, the syntax highlighting
library. My main issue with that library is that themes have to be in Sublime
Text theme format which leaves a lot of nice light/dark themes on the table. My
current approach is a tree-sitter based
<a href="https://github.com/matze/tree-painter">library</a> that bundles a bunch of
tree-sitter grammars and uses <a href="https://github.com/helix-editor/helix">helix</a>
themes to highlight the parsed names. While it works alright, distributing it as
a crate is a pain in the ass because only a fraction keeps publishing updated
grammars on crates.io. So, next idea is perhaps bundling it via Git submodules.
Let’s see.</p>
Yet another pastebin2022-06-09T00:00:00+00:002022-06-09T00:00:00+00:00
Unknown
https://bloerg.net/posts/yet-another-pastebin/<p><a href="https://en.wikipedia.org/wiki/Pastebin">Pastebins</a> are the next step in the
evolution of a software developer, right after finishing hello worlds and static
site generators. They are limited terms of features (or not … ahem) but require
some form of dynamisms in order to receive and store user input and make it
available upon request. Of course, everyone has different ideas what a pastebin
should do and in what language it should be written. And because I am in no way
different, I had to write my own: the
<a href="https://github.com/matze/wastebin">wastebin</a> pastebin that ticks the following
boxes:</p>
<ul>
<li>Written in Rust for ease of deployment.</li>
<li>SQLite instead of a full-fledged database server or flat files.</li>
<li>Paste expiration.</li>
<li>Minimalist appearance.</li>
<li>Syntax highlighting.</li>
<li>Line numbers.</li>
</ul>
<p><a href="https://github.com/WantGuns/bin">bin</a> – from which wastebin takes huge
inspiration in terms of UI – was almost there but the lack of expiration and
flat-file storage was a no-go. Moreover, I sincerely think
<a href="https://github.com/tokio-rs/axum">axum</a> has a more solid foundation than
Rocket. Enough reasons to do it myself.</p>
Serve static content with axum2022-03-01T00:00:00+00:002022-03-01T00:00:00+00:00
Unknown
https://bloerg.net/posts/serve-static-content-with-axum/<p>One of Rust’s nice properties is producing statically linked binaries making
deployment simple and straightforward. In some cases this is not enough and
additional data is required for proper function, for example static data for web
servers. With dependencies such as <code>include_dir</code> and <code>mime_guess</code> this is a
piece of cake to integrate into <code>axum</code> though.</p>
<p>Using <code>include_dir</code> we first declare variable that represents the data currently
located in the <code>static</code> directory:</p>
<pre data-lang="rust" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-rust "><code class="language-rust" data-lang="rust"><span style="color:#72ab00;">use </span><span>include_dir::{include_dir, Dir};
</span><span>
</span><span style="color:#668f14;">static </span><span style="color:#b3933a;">STATIC_DIR</span><span>: Dir<'</span><span style="color:#72ab00;">_</span><span>> </span><span style="color:#72ab00;">= </span><span style="color:#a2a001;">include_dir!</span><span>(</span><span style="color:#d07711;">"$CARGO_MANIFEST_DIR/static"</span><span>);
</span></code></pre>
<p>Now we define the static data route, passing <code>*path</code> to denote we want to match
the entire remaining path.</p>
<pre data-lang="rust" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-rust "><code class="language-rust" data-lang="rust"><span>#[</span><span style="color:#5597d6;">tokio</span><span>::</span><span style="color:#5597d6;">main</span><span>]
</span><span>async </span><span style="color:#668f14;">fn </span><span style="color:#c23f31;">main</span><span>() -> </span><span style="color:#a2a001;">Result</span><span><(), </span><span style="color:#a2a001;">Box</span><span><dyn std::error::Error>> {
</span><span> </span><span style="color:#668f14;">let</span><span> app </span><span style="color:#72ab00;">= </span><span>axum::Router::new()
</span><span> .</span><span style="color:#b39f04;">route</span><span>(</span><span style="color:#d07711;">"/static/*path"</span><span>, </span><span style="color:#b39f04;">get</span><span>(static_path));
</span><span>
</span><span> </span><span style="color:#668f14;">let</span><span> addr </span><span style="color:#72ab00;">= </span><span>std::net::SocketAddr::from(([</span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">0</span><span>], </span><span style="color:#b3933a;">3000</span><span>));
</span><span>
</span><span> axum::Server::bind(</span><span style="color:#72ab00;">&</span><span>addr)
</span><span> .</span><span style="color:#b39f04;">serve</span><span>(app.</span><span style="color:#b39f04;">into_make_service</span><span>())
</span><span> .await</span><span style="color:#72ab00;">?</span><span>;
</span><span>
</span><span> </span><span style="color:#a2a001;">Ok</span><span>(())
</span><span>}
</span></code></pre>
<p>Note that we cannot use the newly added typed path functionality in
<code>axum-extra</code>. Now onto the actual route handler:</p>
<pre data-lang="rust" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-rust "><code class="language-rust" data-lang="rust"><span>async </span><span style="color:#668f14;">fn </span><span style="color:#c23f31;">static_path</span><span>(Path(</span><span style="color:#5597d6;">path</span><span>): Path<</span><span style="color:#a2a001;">String</span><span>>) -> impl IntoResponse {
</span><span> </span><span style="color:#668f14;">let</span><span> path </span><span style="color:#72ab00;">=</span><span> path.</span><span style="color:#b39f04;">trim_start_matches</span><span>(</span><span style="color:#d07711;">'/'</span><span>);
</span><span> </span><span style="color:#668f14;">let</span><span> mime_type </span><span style="color:#72ab00;">= </span><span>mime_guess::from_path(path).</span><span style="color:#b39f04;">first_or_text_plain</span><span>();
</span><span>
</span><span> </span><span style="color:#72ab00;">match </span><span style="color:#b3933a;">STATIC_DIR</span><span>.</span><span style="color:#b39f04;">get_file</span><span>(path) {
</span><span> </span><span style="color:#a2a001;">None </span><span style="color:#72ab00;">=> </span><span>Response::builder()
</span><span> .</span><span style="color:#b39f04;">status</span><span>(StatusCode::</span><span style="color:#b3933a;">NOT_FOUND</span><span>)
</span><span> .</span><span style="color:#b39f04;">body</span><span>(body::boxed(Empty::new()))
</span><span> .</span><span style="color:#b39f04;">unwrap</span><span>(),
</span><span> </span><span style="color:#a2a001;">Some</span><span>(file) </span><span style="color:#72ab00;">=> </span><span>Response::builder()
</span><span> .</span><span style="color:#b39f04;">status</span><span>(StatusCode::</span><span style="color:#b3933a;">OK</span><span>)
</span><span> .</span><span style="color:#b39f04;">header</span><span>(
</span><span> header::</span><span style="color:#b3933a;">CONTENT_TYPE</span><span>,
</span><span> HeaderValue::from_str(mime_type.</span><span style="color:#b39f04;">as_ref</span><span>()).</span><span style="color:#b39f04;">unwrap</span><span>(),
</span><span> )
</span><span> .</span><span style="color:#b39f04;">body</span><span>(body::boxed(Full::from(file.</span><span style="color:#b39f04;">contents</span><span>())))
</span><span> .</span><span style="color:#b39f04;">unwrap</span><span>(),
</span><span> }
</span><span>}
</span></code></pre>
<p>As you can see we first strip the initial slash and then use the <code>mime_guess</code>
crate to guess a MIME type from it. If we are not able to do so, just assume
<code>text/plain</code> however wrong that is. Then we try to locate the file path and
either return a 404 or a 200 with the actual file contents. Easy as pie.</p>
Another interactive Git tool2021-05-10T00:00:00+00:002021-05-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/another-interactive-git-tool/<p><a href="https://bloerg.net/posts/a-git-cherry-pick-text-user-interface/">Eight years ago</a>
(ouch …) I published <a href="http://github.com/matze/git-pick-from">a little tool</a> to
interactively cherry-pick commits from a branch. It’s still useful to me but
ever so often I forget about the urwid dependency when re-installing it. Yada
yada, so of course team Rewrite-in-Rust <a href="https://github.com/matze/git-utils">strikes
again</a> which means I can simply drop the
statically linked binary into <code>$PATH</code> and not have to worry about Python and
ncurses dependencies anymore.</p>
<p>And for the price of one you now get even two tools. Since I have to create a
lot of feature, bugfix and whatnot branches, deleting them becomes painful as
well even with tab-completion and fzf integration. So, in a similar vein to <code>git pick</code> the <code>git prune-branches</code> (<code>git prune</code> is a plumbing command to remove
unreachable objects from the Git database) allows you to select a bunch of
branches interactively and delete them in one go.</p>
inkdrop: fun with points and lines2021-03-30T00:00:00+00:002021-03-30T00:00:00+00:00
Unknown
https://bloerg.net/posts/inkdrop-fun-with-points-and-lines/<p>A friend of mine asked me to join him in a small side project that combines both
art, programming and hardware. The latter is still in the design phase, so let
me tell you about the art and programming part first. The principal idea is to
generate art (random, really generative or from existing data) and let it being
drawn in real life by a <em>machine</em>. We wanted to start simple and just process
existing bitmap images and turn them into nice looking paths that can be drawn
by the “machine”.</p>
<p>My initial idea was to generate a path from a random walk with a probability
density that matches the surrounding pixel density of the current position.
While that works in a way, the result is hardly legible. After some researching
we came to the conclusion to do what in general is known as <a href="https://www2.oberlin.edu/math/faculty/bosch/making-tspart-page.html">TSP art</a>.</p>
<h3 id="tsp-art">TSP art</h3>
<p>TSP as in the Travelling Salesman Problem is a two-step problem. In a first
step, for given number of points, we find a set of points that approximates the
pixel density of the input image. In a second step, we determine a path that
connects all points exactly once.</p>
<h4 id="sampling-points">Sampling points</h4>
<p>Because (photographic) image formation is a stochastic process one can do the
reverse and use the brightness of each input image pixel as the probability that
light hit the sensor. Since we want to have more points for darker areas, we
have to use the inverse though. That and simple rejection sampling leads us to a
basic point sampling algorithm:</p>
<ol>
<li>If we have enough points stop.</li>
<li>Compute a random position (x, y) in [0, width; 0, height].</li>
<li>Read the pixel value at (x, y) in the input image, normalize it to [0, 1] and
invert it.</li>
<li>Compute a random brightness value between [0, 1].</li>
<li>If the random brightness value is larger than the pixel value accept it.</li>
</ol>
<p>This works wonderfully but due to the stochastic nature requires <em>many</em> samples
which is detrimental to the path finding step. In the following image you can
see the result for 10000 and 100000 points:</p>
<figure>
<img src="/images/2021-03-30/nofretete-random-samples-10000-100000.png" />
</figure>
<p>Luckily there are various approaches to approximate the density with less points
the same way a human would draw an image using a pencil.</p>
<h4 id="weighted-voronoi-stippling">Weighted Voronoi stippling</h4>
<p><a href="https://www.cs.ubc.ca/labs/imager/tr/2002/secord2002b/secord.2002b.pdf">Weighted Voronoi Stippling</a> is such an approach. The basic idea in layman’s
terms is as follows:</p>
<ol>
<li>Sample a set of points randomly or use the algorithm from above.</li>
<li>Compute the <a href="https://en.wikipedia.org/wiki/Voronoi_diagram">Voronoi diagram</a> for the point set.</li>
<li>For each Voronoi cell move its central point to the darkest area.</li>
<li>Continue until you are satisfied.</li>
</ol>
<p>The following image shows you the result for 10000 and 30000 points and 5 as
well as 100 iterations:</p>
<figure>
<img src="/images/2021-03-30/nofretete-voronoi-10000-5-30000-100.png" />
</figure>
<p>Note though that there are no major improvements past the 50 iterations mark.</p>
<h4 id="solving-the-travelling-salesman-problem">Solving the Travelling Salesman Problem</h4>
<p>Now that we have a good set of points we want to connect them and ideally visit
each point only once. Sounds familiar? Yes, it’s the infamous Travelling
Salesman Problem which is NP-complete, meaning you won’t find the best solution
in a reasonable amount of time for a larger number of points. So again a bit of
research and the most straightforward algorithm is again two-fold, first we
compute an initial path using the <a href="https://en.wikipedia.org/wiki/Nearest_neighbour_algorithm">Nearest Neighbor algorithm</a> and then
iteratively improve that path using the <a href="https://en.wikipedia.org/wiki/2-opt">2-opt algorithm</a>. Here are two
results for 10000 and 30000 points respectively:</p>
<figure>
<img src="/images/2021-03-30/nofretete-path-10000-30000.png" />
</figure>
<h3 id="inkdrop">inkdrop</h3>
<p>All of the images have generated by a small Rust tool called <a href="https://github.com/matze/inkdrop">inkdrop</a>. It is
split into a core library that implements the algorithms as well as some data
structures and two binaries to drive them via command line or a GTK 4 user
interface. The primary output data format is SVG which is super simple and
matches the problem very well. To generate Voronoi diagrams I used the wonderful
Voronator library which wraps the Delaunator library I tried to use before but
gave me too many headaches.</p>
<p>Long-time readers will know that GTK is nothing new for me but GTK 4 is. And I
must admit, I love the on-going work of the <a href="https://gtk-rs.org/">Gtk-rs</a> team that provides usable
GTK 4 bindings at this early stage as well as the GTK team itself. GTK 4 finally
improves quite a few pain points for me and surprised me positively with the
addition of layout managers and declarative property bindings. This allows one
to write even less code and declare more UI facts in the builder file itself.
Neat.</p>
<figure>
<img src="/images/2021-03-30/ui.png" />
</figure>
<p>So, here is part one of our generative art adventure.</p>
<p><strong>Update</strong>: The Rust bindings to GTK4 are now stable. Yay!</p>
Fixing up Stoneshard dynamic link issues2021-02-27T00:00:00+00:002021-02-27T00:00:00+00:00
Unknown
https://bloerg.net/posts/fixing-up-stoneshard/<p><a href="https://stoneshard.com/">Stoneshard</a> is a turn-based RPG with nice 2D pixel art and an elaborate
role-playing system. Unfortunately, the Linux support is a bit messy and if you
happen to play it on Ubuntu 20.04, it will tell you it cannot find
<code>libcrypto.so.1.0.0</code>. Common advice is to symlink <code>libcrypto.so.1.1.0</code> however
that is neither a good idea nor even necessary because the game does not even
depend on it. Hence, the cleaner way is just to patch the game binary, for
example using <a href="https://github.com/NixOS/patchelf">patchelf</a>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ git clone https://github.com/NixOS/patchelf && cd patchelf
</span><span>$ ./bootstrap.sh && ./configure && make && sudo make install
</span><span>$ patchelf --remove-needed libcrypto.so.1.0.0 <path-to-stoneshard>/game/StoneShard
</span><span>$ patchelf --remove-needed libssl.so.1.0.0 <path-to-stoneshard>/game/StoneShard
</span></code></pre>
<p>With that done, you can run the game just fine. I still wonder what the crappy
build system of that game is doing though …</p>
A waste calendar and map2021-02-18T00:00:00+00:002021-02-18T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-waste-calendar-and-map/<p>As we all know public services are often underfunded or public websites more
often than not of questionable usability. The <a href="https://web6.karlsruhe.de/service/abfall/akal/akal.php">waste calendar</a> for the
municipality of Karlsruhe is somewhere in between. It is not bad but not of much
help for your fellow scavengers who like to re-use thrown away goods. For the
past three years, I scraped that site with a small Python <a href="https://github.com/matze/abfallkalender/blob/8db01ac7f288d657897a0bd5e98e325685002884/scrape.py">script</a> and dumped
a CSV file that people could use to plan their hunts. This worked well enough
but I always only updated it begrudgingly because I was asked to do it rather
than because I wanted to do it.</p>
<p>But <em>I am</em> motivated to hone my programming skills, so I took this little
project, <a href="https://github.com/matze/abfallkalender">rewrote it in Rust</a>, extracted road and date data and dumped
everything into a small <a href="https://matze.github.io/abfallkalender/">static site</a> based on Vue and Leaflet.js. So, if you
want to schedule your next hunt or find out where to play Pokémon GO in
Karlsruhe without being disturbed, here is your chance.</p>
<figure>
<a href="https://matze.github.io/abfallkalender/">
<img src="https://camo.githubusercontent.com/17a154ced85cf6ce349ce0dec3f76d1980fa134e1183bf6f6306a37ad67e287f/68747470733a2f2f6d61747a652e6769746875622e696f2f616266616c6c6b616c656e6465722f696d6167652e706e67" />
</a>
</figure>
<p>P.S.: thanks Karlsruhe for that splendid API /s</p>
Going Zola2021-02-17T00:00:00+00:002021-02-17T00:00:00+00:00
Unknown
https://bloerg.net/posts/going-zola/<p>As easily witnessed, content frequency of the blog is at an all-time low. So
what’s better than increasing it by writing about switching from Jekyll to
<a href="https://www.getzola.org">Zola</a>? Why, you ask? First, Jekyll is the only reason I have Ruby and gazillion of
Jekyll dependencies installed even though I don’t understand a lick of the
language. Second, every time there is a Jekyll update, I have to do something
about some plugin breakage or live with some new warnings about this and that.
Third, considering the application scope of a static site generator, it’s
incredibly slow.</p>
<p>Actually, there was no urgency but I read about Zola and was hooked because it’s
supposed to be fast (it is, taking a mere 100 ms to re-generate this blog) and
it uses the same <a href="https://tera.netlify.app">templating technology</a> I used for <a href="https://github.com/matze/splat">splat</a>. I won’t bore you
with the conversion details but just want to point out <a href="https://gist.github.com/matze/7d899f6a8caebcd43eb9198167ac77ac">this rewrite script</a>
which converts the Jekyll front matter which is written in YAML to TOML which is
required by Zola.</p>
<p>All in all it was a straightforward conversion and I hope Zola will backing this
blog for the next nine years as Jekyll did in the past.</p>
Splat2021-02-08T00:00:00+00:002021-02-08T00:00:00+00:00
Unknown
https://bloerg.net/posts/splat/<p>Static site generators for blogs and similar ad-ladden static “content” is the
hot shit for years now. Probably because it’s a simple enough task to tackle
with even superficial knowledge of the language du jour. Because of that and the
number of people hosting their own photos being in the low single digits, static
site generators for photo galleries are not a hot topic. For years, I have used
<a href="http://sigal.saimon.org/en/latest/">Sigal</a> and even contributed a few patches here and there. Color me surprised
when I updated to the latest version and learned that I had to update my carefully
crafted theme and configuration to continue using that. Well, not with me! I am
stupid enough to use my superficial knowledge of Rust to come up with <a href="https://github.com/matze/splat">splat</a>, an
alternative static site generator that will never break themes. Promised 🤞</p>
ninja backend for coremake2020-12-06T00:00:00+00:002020-12-06T00:00:00+00:00
Unknown
https://bloerg.net/posts/ninja-backend-for-coremake/<p>C and C++ is known for its abysmal dependency and build system situation. For
historic reasons a build system has not been standardized thus low level build
tools such as <code>make</code> and <code>ninja</code> as well high-level build tool generators such
as Autotools, CMake, meson and entire build frameworks like bazel have come and
gone throughout the years.</p>
<p>A lesser known build generator is
<a href="https://github.com/corecodec/coremake">coremake</a>. And lets be honest: it is a
horrible, underdocumented piece of shit written in hard-to-follow spaghetti C. I
would not even call it a build system generator it is more a static generator
tool that reads its own undocumented domain-specific language (which is actually
not that bad because like meson it can only declare dependencies) written in
<code>.proj</code> files and a so-called “platform” file that, amusingly, ends in <code>.build</code>.
That platform file is does not just specify a certain tool
chain and maybe <code>CFLAGS</code> but is a hodge podge of that and instructions how to
generate output. As you can tell from the official repo, insane people have done the
insane and written generator code for Android, <code>make</code>, Visual Studio and
whatnot.</p>
<p>Since, the <code>make</code> backend defaults to recursive <code>Makefile</code>s (booo), I was
tempted to add one more to the whatnots and that is
<a href="https://ninja-build.org">ninja</a>. Some may know it as the default backend of
<a href="https://mesonbuild.com">meson</a> and an optional backend of CMake. Anyhow, on a
scale of 1 (make) to 10 (bazel) in terms of stuff, it ranges somewhere between
0.01 and 0.02 thus all logic <em>must</em> come from a meta build system. And to spare
you the work of figuring this out, <a href="https://github.com/matze/coreninja">here is something for the second
advent</a> which allows you doing this</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ coremake ninja-clang-linux_86_64
</span><span>$ ninja
</span><span>$ ninja -t compdb > compile_commands.json
</span></code></pre>
<p>Adding basic ninja support to coremake was a small adventure. Although ninja’s
description language is super easy to follow, figuring out coremake’s quirks
concerning scoping and path makes you want to puke and I am certain I have only
scratched 5% of the disgusting surface.</p>
Canon EOS RP under Linux2019-04-16T00:00:00+00:002019-04-16T00:00:00+00:00
Unknown
https://bloerg.net/posts/canon-eos-rp-under-linux/<p>Canon shipped their newest and cheapest full frame camera, the Canon EOS RP, end
of February. In their infinite wisdom, they decided that they <em>had to</em> to
develop a new CR3 format for this as well as the previously released EOS R and
M50 cameras. And of course, so far no one was able to <a href="https://github.com/darktable-org/rawspeed/issues/121">reverse-engineer the raw
format</a> entirely, so it
will take a good while until those cameras will be supported by all major open
source raw programs out of the box. The last option is running the Adobe DNG
Converter in a Windows VM or under Wine to convert the proprietary CR3 format
into their (intermediate) DNG format. After a good month, Adobe released a <a href="https://supportdownloads.adobe.com/detail.jsp?ftpID=6641">new
version</a> that
supported the RP … but it stopped working under Wine 🤦. So, whoever stumbles
upon</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Unhandled exception: unimplemented function api-ms-win-core-winrt-error-l1-.GetRestrictedErrorInfo called in 64-bit code
</span></code></pre>
<p>open <code>winecfg</code>, go to Libraries, add two entries for
<code>api-ms-win-core-winrt-error-l1-1-0</code> and deactivate them. The DNG converter will
complain but it will work (much faster than in a Windows 10 VM!) nevertheless.</p>
Time tracking with Ledger2019-03-02T00:00:00+00:002019-03-02T00:00:00+00:00
Unknown
https://bloerg.net/posts/time-tracking-with-ledger/<p>Whoever is reading this blog knows that I use
<a href="/2016/02/22/ledger-import-of-deutsche-bank-data.html">Ledger-likes</a> to track my
finances. Some of you may also know that the currency is just some arbitrary
name for any kind of unit. And a minority of you may also know there is special
support for tracking <em>time</em> in the original Ledger program. This post explains
how I use Ledger and a few Bash aliases to track different activities during a
normal work day.</p>
<p>In a “regular” Ledger file you will find <em>transactions</em> that describe the flow
of a <em>commodity</em> from one (or more) to another <em>account</em> at a certain time.
There is however special support for <a href="https://www.ledger-cli.org/3.0/doc/ledger3.html#Time-Keeping">timelog</a> entries in the ledger program.
They look similar but have special syntax to describe the start and end of a
“transaction”</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>i 2019/03/02 00:30:20 Entertainment:Netflix
</span><span>o 2019/03/02 00:35:20
</span></code></pre>
<p>which basically state what to account from <code>i</code> to <code>o</code>. Suppose you have a
timelog file <code>foo.ledger</code> then <code>ledger -f time.ledger bal</code> would give you
something like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span> 9.07h Work
</span><span> 32.3m Mail
</span><span> 6.9m Admin
</span><span> 8.40h Development
</span><span> 1.69h Entertainment:Netflix
</span><span> 14.2m Drinking
</span><span> 11.9m Meetings
</span><span>--------------------
</span><span> 11.19h
</span></code></pre>
<p>Of course, you can customize the date range and hierarchy depth. Let’s alias
that to</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#b39f04;">alias </span><span style="color:#c23f31;">wasted</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'ledger -f ${TIMELOG} bal -b $(date -dlast-monday +%m/%d) --depth 2'
</span></code></pre>
<p>All good. Now, adding new entries by “punching in” new lines like that above is
more than just cumbersome, it’s something a simple alias could do as well. I
have defined something like</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#b39f04;">alias </span><span style="color:#c23f31;">clock-in</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'echo i $(date +"%Y/%m/%d %H:%M:%S") >> ${TIMELOG}'
</span><span style="color:#b39f04;">alias </span><span style="color:#c23f31;">clock-out</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'echo o $(date +"%Y/%m/%d %H:%M:%S") >> ${TIMELOG}'
</span></code></pre>
<p>where <code>TIMELOG</code> points to the <code>time.ledger</code> file. Once you type</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#5597d6;">$</span><span> clock-in Work:Mail
</span></code></pre>
<p>an appropriate transaction will be made. To check what’s currently going on,
this alias might help:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#b39f04;">alias </span><span style="color:#c23f31;">clock-status</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'[[ $(tail -1 ${TIMELOG} | cut -c 1) == "i" ]] && { echo "Clocked IN to $(tail -1 ${TIMELOG} | cut -d " " -f 4)"; wasted; } || { echo "Clocked OUT"; wasted;}'
</span></code></pre>
<p>That all looks nice and dandy until you realize you don’t remember a particular
activity you want to account you current time on. If you have used any CLI
program you probably hit <kbd>Tab</kbd> more than twice. Wait no more, just define</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span>
</span><span style="color:#668f14;">function </span><span style="color:#c23f31;">_clock_in </span><span>()
</span><span>{
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">cur prev
</span><span> </span><span style="color:#5597d6;">_get_comp_words_by_ref -n</span><span> : cur
</span><span>
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">words</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"$(</span><span style="color:#acb3c2;">cut -d </span><span style="color:#d07711;">' '</span><span style="color:#acb3c2;"> -s -f</span><span style="color:#d07711;"> 4 ${</span><span style="color:#acb3c2;">TIMELOG</span><span style="color:#d07711;">} </span><span style="color:#72ab00;">| </span><span style="color:#acb3c2;">sed </span><span style="color:#d07711;">'/^$/d' </span><span style="color:#72ab00;">| </span><span style="color:#acb3c2;">sort </span><span style="color:#72ab00;">| </span><span style="color:#acb3c2;">uniq</span><span style="color:#d07711;">)"
</span><span> </span><span style="color:#5597d6;">COMPREPLY</span><span style="color:#72ab00;">=</span><span>($(</span><span style="color:#5597d6;">compgen -W </span><span style="color:#d07711;">"${</span><span style="color:#acb3c2;">words</span><span style="color:#d07711;">}"</span><span style="color:#72ab00;"> -- </span><span>${</span><span style="color:#5597d6;">cur</span><span>}))
</span><span> </span><span style="color:#5597d6;">__ltrim_colon_completions </span><span style="color:#d07711;">"${</span><span style="color:#acb3c2;">cur</span><span style="color:#d07711;">}"
</span><span>}
</span><span>
</span><span style="color:#5597d6;">complete -F</span><span> _clock_in clock-in
</span></code></pre>
<p>and you are all set to complete the timelog entry while you clock in. You could
write more elaborate logic in an appropriate scripting language but that’s an
exercise for the reader.</p>
meson and Google Test2018-09-19T00:00:00+00:002018-09-19T00:00:00+00:00
Unknown
https://bloerg.net/posts/meson-and-google-test/<p>I wrote about <a href="/2017/07/14/use-meson.html">meson</a> the awesome build system before. For C-based projects
with many test executables there is nice infrastructure, however many C++
projects probably use <a href="https://github.com/google/googletest">Google Test</a> or <a href="https://github.com/catchorg/Catch2">Catch</a> and a single binary which
runs the entire test suite. This is all nice and dandy with Google Test if you
compile and link the test executable straight from the unit test source files.
If however you build intermediate static libraries for organizational reasons
you will quickly notice that Google Test won’t run anything at all because the
symbols from Google Test itself <a href="https://github.com/google/googletest/issues/481">won’t end up in the final
binary</a> without specifying the
<code>--whole-archive</code> flag. Luckily, meson got the <code>link_whole</code> parameter since
version 0.46, so instead of declaring your static test library as</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>test_lib = static_library('testlib',
</span><span> sources: test_sources,
</span><span> dependencies: [gtest_dep] + build_deps,
</span><span>)
</span><span>
</span><span>test_dep = declare_dependency(
</span><span> link_with: test_lib,
</span><span> dependencies: other_deps,
</span><span>)
</span><span>
</span><span>test_binary = executable('testfoo',
</span><span> sources: ['main.cpp'],
</span><span> dependencies: [test_dep],
</span><span>)
</span></code></pre>
<p>you would change <code>test_dep</code> to</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>test_dep = declare_dependency(
</span><span> link_whole: test_lib,
</span><span> dependencies: other_deps,
</span><span>)
</span></code></pre>
<p>and run your tests as usual.</p>
Moving on to beancount2018-05-14T00:00:00+00:002018-05-14T00:00:00+00:00
Unknown
https://bloerg.net/posts/moving-on-to-beancount/<p>For the past three years I have been recording my finances with tools from the
<a href="/2016-02-22-ledger-import-of-deutsche-bank-data.html">ledger family</a>. While I had no major issues with them, there were a few minor
annoyances, most notably recording capital gains in <em>hledger</em> without a virtual
posting and a lack of a nice visual representation of my finances. I knew about
<a href="http://furius.ca/beancount">beancount</a> but was always a bit sceptical about its data format which is not
really compatible with the other ledger tools. The deciding factor to take the
plunge was the <a href="https://github.com/beancount/fava">fava</a> web interface and the comprehensive inventory system
which makes recording capital gains a breeze.</p>
<h3 id="importing-deutsche-bank-data">Importing Deutsche Bank data</h3>
<p>Moving the ledger data to the beancount format was pretty straightforward using
the <a href="https://github.com/zacchiro/ledger2beancount">ledger2beancount</a> tool. Unfortunately, I had to re-write the import tool
from scratch because beancount does not provide a command similar to <code>hledger csv</code>. On the other hand, it was relatively simple to come up with these little
bits of Python:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">import </span><span>os
</span><span style="color:#72ab00;">import </span><span>re
</span><span style="color:#72ab00;">import </span><span>codecs
</span><span style="color:#72ab00;">import </span><span>csv
</span><span style="color:#72ab00;">import </span><span>datetime
</span><span style="color:#72ab00;">from </span><span>beancount.core </span><span style="color:#72ab00;">import </span><span>number, data, amount
</span><span style="color:#72ab00;">from </span><span>beancount.ingest </span><span style="color:#72ab00;">import </span><span>importer
</span><span>
</span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">Target</span><span>(</span><span style="font-style:italic;color:#b06936;">object</span><span>):
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#b39f04;">__init__</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">account</span><span>, </span><span style="color:#5597d6;">payee</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#5597d6;">narration</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">None</span><span>):
</span><span> </span><span style="color:#5597d6;">self</span><span>.account </span><span style="color:#72ab00;">= </span><span>account
</span><span> </span><span style="color:#5597d6;">self</span><span>.payee </span><span style="color:#72ab00;">= </span><span>payee
</span><span> </span><span style="color:#5597d6;">self</span><span>.narration </span><span style="color:#72ab00;">= </span><span>narration
</span><span>
</span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">DeutscheBankImporter</span><span>(</span><span style="font-style:italic;color:#b06936;">importer.ImporterProtocol</span><span>):
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#b39f04;">__init__</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">account</span><span>, </span><span style="color:#5597d6;">default</span><span>, </span><span style="color:#5597d6;">mapping</span><span>):
</span><span> </span><span style="color:#5597d6;">self</span><span>.account </span><span style="color:#72ab00;">= </span><span>account
</span><span> </span><span style="color:#5597d6;">self</span><span>.default </span><span style="color:#72ab00;">= </span><span>default
</span><span> </span><span style="color:#5597d6;">self</span><span>.mapping </span><span style="color:#72ab00;">= </span><span>mapping
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">identify</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">fname</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span>re.</span><span style="color:#5597d6;">match</span><span>(</span><span style="color:#668f14;">r</span><span style="color:#d07711;">"</span><span style="color:#7c8f4c;">Kontoumsaetze_</span><span style="color:#aeb52b;">\d</span><span style="color:#72ab00;">+</span><span style="color:#7c8f4c;">_</span><span style="color:#aeb52b;">\d</span><span style="color:#72ab00;">+</span><span style="color:#7c8f4c;">_</span><span style="color:#aeb52b;">\d</span><span style="color:#72ab00;">+</span><span style="color:#7c8f4c;">_</span><span style="color:#aeb52b;">\d</span><span style="color:#72ab00;">+</span><span style="color:#aeb52b;">.</span><span style="color:#7c8f4c;">csv</span><span style="color:#d07711;">"</span><span>,
</span><span> os.path.</span><span style="color:#5597d6;">basename</span><span>(fname.name))
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">file_account</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">fname</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#5597d6;">self</span><span>.account
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">extract</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">fname</span><span>):
</span><span> fp </span><span style="color:#72ab00;">= </span><span>codecs.</span><span style="color:#5597d6;">open</span><span>(fname.name, </span><span style="color:#d07711;">'r'</span><span>, </span><span style="color:#d07711;">'iso-8859-1'</span><span>)
</span><span> lines </span><span style="color:#72ab00;">= </span><span>fp.</span><span style="color:#5597d6;">readlines</span><span>()
</span><span>
</span><span> </span><span style="color:#7f8989;"># drop top and bottom stuff
</span><span> lines </span><span style="color:#72ab00;">= </span><span>lines[</span><span style="color:#b3933a;">5</span><span>:]
</span><span> lines </span><span style="color:#72ab00;">= </span><span>lines[:</span><span style="color:#72ab00;">-</span><span style="color:#b3933a;">1</span><span>]
</span><span> entries </span><span style="color:#72ab00;">= </span><span>[]
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">fix_decimals</span><span>(</span><span style="color:#5597d6;">s</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span>s.</span><span style="color:#5597d6;">replace</span><span>(</span><span style="color:#d07711;">'.'</span><span>, </span><span style="color:#d07711;">''</span><span>).</span><span style="color:#5597d6;">replace</span><span>(</span><span style="color:#d07711;">','</span><span>, </span><span style="color:#d07711;">'.'</span><span>)
</span><span>
</span><span> </span><span style="color:#72ab00;">for </span><span>index, row </span><span style="color:#72ab00;">in </span><span style="color:#b39f04;">enumerate</span><span>(csv.</span><span style="color:#5597d6;">reader</span><span>(lines, </span><span style="color:#5597d6;">delimiter</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">';'</span><span>)):
</span><span> meta </span><span style="color:#72ab00;">= </span><span>data.</span><span style="color:#5597d6;">new_metadata</span><span>(fname.name, index)
</span><span> date </span><span style="color:#72ab00;">= </span><span>datetime.datetime.</span><span style="color:#5597d6;">strptime</span><span>(row[</span><span style="color:#b3933a;">0</span><span>], </span><span style="color:#d07711;">'</span><span style="color:#aeb52b;">%d</span><span style="color:#d07711;">.</span><span style="color:#aeb52b;">%m</span><span style="color:#d07711;">.</span><span style="color:#aeb52b;">%Y</span><span style="color:#d07711;">'</span><span>).</span><span style="color:#5597d6;">date</span><span>()
</span><span> desc </span><span style="color:#72ab00;">= </span><span>row[</span><span style="color:#b3933a;">4</span><span>]
</span><span> payee </span><span style="color:#72ab00;">= </span><span>row[</span><span style="color:#b3933a;">3</span><span>]
</span><span> credit </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">fix_decimals</span><span>(row[</span><span style="color:#b3933a;">15</span><span>]) </span><span style="color:#72ab00;">if </span><span>row[</span><span style="color:#b3933a;">15</span><span>] </span><span style="color:#72ab00;">!= </span><span style="color:#d07711;">'' </span><span style="color:#72ab00;">else </span><span style="color:#b3933a;">None
</span><span> debit </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">fix_decimals</span><span>(row[</span><span style="color:#b3933a;">16</span><span>]) </span><span style="color:#72ab00;">if </span><span>row[</span><span style="color:#b3933a;">16</span><span>] </span><span style="color:#72ab00;">!= </span><span style="color:#d07711;">'' </span><span style="color:#72ab00;">else </span><span style="color:#b3933a;">None
</span><span> currency </span><span style="color:#72ab00;">= </span><span>row[</span><span style="color:#b3933a;">17</span><span>]
</span><span> account </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">self</span><span>.default
</span><span> num </span><span style="color:#72ab00;">= </span><span>number.</span><span style="color:#5597d6;">D</span><span>(credit </span><span style="color:#72ab00;">if </span><span>credit </span><span style="color:#72ab00;">else </span><span>debit)
</span><span> units </span><span style="color:#72ab00;">= </span><span>amount.</span><span style="color:#5597d6;">Amount</span><span>(num, currency)
</span><span>
</span><span> </span><span style="color:#72ab00;">for </span><span>p, t </span><span style="color:#72ab00;">in </span><span style="color:#5597d6;">self</span><span>.mapping.</span><span style="color:#5597d6;">items</span><span>():
</span><span> </span><span style="color:#72ab00;">if </span><span>p </span><span style="color:#72ab00;">in </span><span>desc:
</span><span> account </span><span style="color:#72ab00;">= </span><span>t.account
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>t.narration:
</span><span> desc </span><span style="color:#72ab00;">= </span><span>t.narration
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>t.payee:
</span><span> payee </span><span style="color:#72ab00;">= </span><span>t.payee
</span><span>
</span><span> frm </span><span style="color:#72ab00;">= </span><span>data.</span><span style="color:#5597d6;">Posting</span><span>(</span><span style="color:#5597d6;">self</span><span>.account, units, </span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#b3933a;">None</span><span>)
</span><span> to </span><span style="color:#72ab00;">= </span><span>data.</span><span style="color:#5597d6;">Posting</span><span>(account, </span><span style="color:#72ab00;">-</span><span>units, </span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#b3933a;">None</span><span>, </span><span style="color:#b3933a;">None</span><span>)
</span><span> txn </span><span style="color:#72ab00;">= </span><span>data.</span><span style="color:#5597d6;">Transaction</span><span>(meta, date, </span><span style="color:#d07711;">"*"</span><span>, payee, desc,
</span><span> data.</span><span style="color:#5597d6;">EMPTY_SET</span><span>, data.</span><span style="color:#5597d6;">EMPTY_SET</span><span>, [frm, to])
</span><span>
</span><span> entries.</span><span style="color:#5597d6;">append</span><span>(txn)
</span><span>
</span><span> </span><span style="color:#72ab00;">return </span><span>entries
</span></code></pre>
<p>that you would plug in to your import config like this:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span>mappings </span><span style="color:#72ab00;">= </span><span>{
</span><span> </span><span style="color:#d07711;">'Salary'</span><span>:
</span><span> </span><span style="color:#5597d6;">Target</span><span>(</span><span style="color:#d07711;">'Assets:Income'</span><span>, </span><span style="color:#d07711;">'Foo Company'</span><span>),
</span><span> </span><span style="color:#d07711;">'Walmart'</span><span>:
</span><span> </span><span style="color:#5597d6;">Target</span><span>(</span><span style="color:#d07711;">'Expenses:Food:Groceries'</span><span>),
</span><span>}
</span><span style="color:#5597d6;">CONFIG </span><span style="color:#72ab00;">= </span><span>[
</span><span> </span><span style="color:#5597d6;">DeutscheBankImporter</span><span>(</span><span style="color:#d07711;">'Assets:Checking'</span><span>, </span><span style="color:#d07711;">'Expenses:ReplaceMe'</span><span>, mappings)
</span><span>]
</span></code></pre>
<p>Yes, that’s my answer to this statement from the official documentation:</p>
<blockquote>
<p>My standard answer is that while it would be fun to have [automatic
categorization], if you have a text editor with account name completion
configured properly, it’s a breeze to do this manually and you don’t really
need it.</p>
</blockquote>
<p>On to the next years …</p>
Trying vcpkg2018-04-25T00:00:00+00:002018-04-25T00:00:00+00:00
Unknown
https://bloerg.net/posts/trying-vcpkg/<p>Developing C and C++ projects in a cross-platform manner is not only difficult
because of different system APIs but also because dependencies have to be
tracked, installed and integrated into the build system <em>du jour</em>. While it is
largely a solved problem on Linux thanks to system package managers and de facto
standards such as <code>pkg-config</code>, the situation on Windows is tricky because both
are missing. Rust and Go easily circumvent this situation by building static
binaries from source using their integrated package managers. In a similar
manner a Microsoft team tries to combine package management and build
integration on Linux, Mac and Windows with their open source <a href="https://github.com/Microsoft/vcpkg">vcpkg</a> package
manager.</p>
<p>The main idea is simple: 1) provide a repository with build and dependency
descriptions for a wide range of libraries, 2) a CLI tool to actually build said
libraries locally and 3) means to make use of these libraries. Each of these
points has its pros and cons you can see when trying to build the following C
toy example on Linux which depends on the GLib utility library:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#72ab00;">#include </span><span style="color:#d07711;"><glib.h>
</span><span>
</span><span style="color:#668f14;">int
</span><span style="color:#c23f31;">main </span><span>(</span><span style="color:#668f14;">int </span><span style="color:#5597d6;">argc</span><span>, </span><span style="color:#668f14;">char </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">argv</span><span>[])
</span><span>{
</span><span> </span><span style="color:#5597d6;">g_print </span><span>(</span><span style="color:#d07711;">"Hello world</span><span style="color:#aeb52b;">\n</span><span style="color:#d07711;">"</span><span>);
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#b3933a;">0</span><span>;
</span><span>}
</span></code></pre>
<p>With vcpkg you first have to clone the repository and then build the <code>vcpkg</code>
binary. Easier said than done because the binary requires g++-7 to build. You
then use </p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>vcpkg install glib
</span></code></pre>
<p>to build GLib and all its dependencies which works well but has one big caveat:
Using so-called <code>CONTROL</code> files, maintainers specify the version of the library
and all their dependencies. Unfortunately, build dependencies are <em>not</em>
versioned but taken from the current state of the repository. This means you can
only pin the version of library A by not upgrading the repository therefore you
are unable to upgrade to a newer version of library B. In the end, <code>vcpkg</code> is
just like a system package manager and we are nowhere near closer to the Rust
build story.</p>
<p>Now, how do we use our library? Well there’s the first problem. At the moment,
the only officially supported way is to write a CMake build script such as this</p>
<pre data-lang="cmake" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cmake "><code class="language-cmake" data-lang="cmake"><span style="color:#b39f04;">cmake_minimum_required</span><span>(</span><span style="color:#5597d6;">VERSION </span><span>3.0)
</span><span style="color:#b39f04;">project</span><span>(hello)
</span><span>
</span><span style="color:#b39f04;">find_path</span><span>(GLIB_INCLUDE_DIR glib.h)
</span><span style="color:#b39f04;">find_library</span><span>(GLIB glib REQUIRED)
</span><span>
</span><span style="color:#b39f04;">add_executable</span><span>(hello hello.c)
</span><span style="color:#b39f04;">target_link_libraries</span><span>(hello </span><span style="color:#5597d6;">PRIVATE </span><span>${</span><span style="color:#5597d6;">GLIB_LIBRARY</span><span>})
</span><span style="color:#b39f04;">target_include_directories</span><span>(hello ${</span><span style="color:#5597d6;">GLIB_INCLUDE_DIR</span><span>})
</span></code></pre>
<p>and passing their toolchain file to the configure process of CMake</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>cmake <builddir> . -DCMAKE_TOOLCHAIN_FILE=<vcpkgroot>/scripts/buildsystems/vcpkg.cmake
</span></code></pre>
<p>For projects that have native CMake support you can use the <code>find_package</code>
command but as you can see we have to rely on the <code>find_library</code> and <code>find_path</code>
commands for GLib. Except, it does not work. CMake is unable to find neither
GLib <code>sqlite3</code> using <code>find_package</code> as outlined in the documentation. Could we
at least build something by hand? Kind of. All libraries and header files are
installed under <code><vcpkgroot>/installed/x64-linux</code> and the various
sub-directories. However, you have to hardcode the paths because <code>pkg-config</code>
files are not installed thus its worse than using system development libraries.</p>
<p>Because of these issues, I would not consider <code>vcpkg</code> to be an alternative
package and build manager for Linux at the time of writing. There are some good
ideas but <a href="https://spack.io">spack</a> pretty much does the same for a while now
and they gave way more time thinking into their product. Let’s hope it will get
better.</p>
Using a Focusrite 18i8 under Linux2018-02-01T00:00:00+00:002018-02-01T00:00:00+00:00
Unknown
https://bloerg.net/posts/using-a-focusrite-18i8-under-linux/<p>Focusrite’s <a href="https://focusrite.de/scarlett-range">Scarlett</a> USB audio interfaces
are cost-effective units with a wide range of input and output options. Out of
the box they can be used by most digital audio workstations on Linux because
they are USB class-compliant. However, besides the entry-level models (Solo, 2i2
and 2i4), you will certainly miss a few features for recording that just cannot
be accessed due to a lack hardware controls. For these you need to use the
Focusrite Control software which – as you can imagine – is available only for
Mac and Windows. <em>Fortunately</em>, there is a device specific ALSA driver for the
Scarlett devices that exposes mix controls to set gains on the different ins and
outs as well as controls to enable Hi-Z or Pad modes on certain inputs.
<em>Unfortunately</em>, using <code>alsamixer</code> to set these values is a huge PITA.</p>
<p>Now <a href="https://gareus.org/">Robin Gareus</a>, a very active contributor to Ardour,
JACK and LV2 made the <a href="https://github.com/x42/scarlett-mixer">scarlett-mixer</a>
for his Scarlett 18i6 that allows one to control all the parameters with an
easy-to-use graphical user interface. Now, I counted one and one together, made
the appropriate changes and added support for the 18i8. So, no longer do I need
to resort to the borderline crazy Alsa Json Gateway software 🙌</p>
OpenCL resource tracking2018-01-25T00:00:00+00:002018-01-25T00:00:00+00:00
Unknown
https://bloerg.net/posts/opencl-resource-tracking/<p>I wrote about the woes I had with the abysmal OpenCL implementations by both AMD
and NVIDIA. Now, it’s not always the fault of others and I recently had to debug
a GPU memory leak originating from our own software. The first thing I needed
to indentify was if all OpenCL resources were referenced and dereferenced
equally. However, the entire system is (as usual I suppose) a dynamic pile of
layers of abstractions, so simple reasoning from the code was out of the
question rather quickly. I remembered the <a href="https://bloerg.net/posts/opencl-resource-tracking/20137/06/06/tracking-gobjects.html">GObject tracking
library</a> from which I took the interception
and backtrace bits, re-implemented the OpenCL calls dealing with resource
allocation and unreferencing to make the <code>LD_PRELOAD</code> library
<a href="https://github.com/matze/ocl-stat">libocl-stat.so</a>. Usage is similar to the
GObject tracking library <code>gobject-list</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ LD_PRELOAD=/path/to/libocl-stat.so ./app
</span></code></pre>
<p>with an example output like (now):</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>OpenCL objects alive
</span><span>====================
</span><span>Contexts 1/1
</span><span>Command queues 0/1
</span><span>Buffers 0/9
</span><span>Samplers 0/0
</span><span>Kernels 3/11
</span><span>
</span><span>Memory leaks
</span><span>============
</span><span>Leaking 0 B
</span></code></pre>
Moving music content2017-08-28T00:00:00+00:002017-08-28T00:00:00+00:00
Unknown
https://bloerg.net/posts/moving-music-content/<p>I decided to move all music and bass related content to its own platform over at
<a href="http://music.bloerg.net">music.bloerg.net</a> because it does not really fit the
technical stuff here. No redirects were set up, so if you happened to have
linked there … update your links accordingly.</p>
Converting PDF pages to a single image2017-08-22T00:00:00+00:002017-08-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/converting-pdf-pages-to-a-single-image/<p>I am always amazed how far one can get by chaining the output of simple programs
to create something bigger. This time, I had to convert the pages of a PDF file
to a single PNG thumbnail image. As I learned in five minutes this is super
simple with <code>pdftoppm</code>, Imagemagick’s <code>convert</code> tool and a <code>Makefile</code> to hook it
up together:</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#5597d6;">%</span><span style="color:#c23f31;">.png</span><span style="color:#72ab00;">: </span><span style="color:#acb3c2;">%</span><span style="color:#d07711;">.pdf
</span><span> </span><span style="color:#5597d6;">pdftoppm -png $<</span><span> tmp
</span><span> </span><span style="color:#5597d6;">convert</span><span> +append tmp-</span><span style="color:#72ab00;">*</span><span style="color:#5597d6;"> -resize</span><span> x320</span><span style="color:#5597d6;"> -unsharp</span><span> 0x1 </span><span style="color:#5597d6;">$@
</span><span> </span><span style="color:#5597d6;">rm</span><span> tmp-</span><span style="color:#72ab00;">*
</span></code></pre>
<p>What I do here is calling <code>pdftoppm</code> to generate numbered PNG files with a
<code>tmp-</code> prefix. <code>convert</code> reads them, appends them horizontally (denoted by the +
sign), resizes the final image to a height of 320 pixels and filters the output
with an unsharp mask to “recover” some of the lost sharpness.</p>
<p>To make the pages stand out you can also add a black single pixel frame and a
white separation border with</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#5597d6;">%</span><span style="color:#c23f31;">.png</span><span style="color:#72ab00;">: </span><span style="color:#acb3c2;">%</span><span style="color:#d07711;">.pdf
</span><span> </span><span style="color:#5597d6;">convert $<</span><span>-</span><span style="color:#72ab00;">*</span><span>.png</span><span style="color:#5597d6;"> -resize</span><span> x320</span><span style="color:#5597d6;"> -unsharp</span><span> 0x1</span><span style="color:#5597d6;"> -bordercolor</span><span> black</span><span style="color:#5597d6;"> -border</span><span> 1x1</span><span style="color:#5597d6;"> -bordercolor</span><span> white</span><span style="color:#5597d6;"> -border</span><span> 2x2 +append </span><span style="color:#5597d6;">$@
</span></code></pre>
Beamer, movies and Linux2017-08-17T00:00:00+00:002017-08-17T00:00:00+00:00
Unknown
https://bloerg.net/posts/beamer-movies-and-linux/<p>tl;dr: On a stock Ubuntu 16.04. system build your presentation with pdfLaTeX or
LuaTeX and use Okular to display it.</p>
<p>TikZ animations are a great way to illustrate dynamic processes in your Beamer
presentations but are not suited for movies. So far, presenting movies on Linux
has been pretty dull but eventually I figured out what works (and what not).
First of all you, need to include</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#72ab00;">\usepackage</span><span>{</span><span style="color:#a2a001;">multimedia</span><span>}
</span></code></pre>
<p>in your preamble which gives you the <code>\movie</code> command in order to include a
<em>reference</em> to the movie file and a placeholder text/image/whatever that is
shown as long as the movie is not played:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\movie</span><span>[width=9.6cm, height=5.4cm]{
</span><span> </span><span style="color:#b39f04;">\includegraphics</span><span>[</span><span style="color:#5597d6;">width</span><span>=9.6cm, height=5.4cm]{still}
</span><span>}{movie.wmv}
</span></code></pre>
<p>In most cases, the first frame of the movie would be a good canditate as a
placeholder. To get the first frame you could use ffmpeg:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#5597d6;">ffmpeg -i</span><span> movie.wmv</span><span style="color:#5597d6;"> -vframes</span><span> 1</span><span style="color:#5597d6;"> -f</span><span> image2 foo.jpg
</span></code></pre>
<p>Now, the first hurdle is to compile the presentation and unfortunately you will
find out that XeTeX does <em>not</em> work with the <code>multimedia</code> package. So, either
compile with LuaTeX if you want to use system fonts or compile with regular old pdfLaTeX.</p>
<p>To display your presentation you <em>could</em> use Evince but it will display the
animation <em>only</em> in windowed mode, in the presentation mode nothing happens for
me. KDE’s Okular has no such limitation and works fine with a PDF containing a
movie. Note though that the movies are not baked into PDF but linked to. This
means, you must deliver all files side-by-side.</p>
Redesign2017-08-16T00:00:00+00:002017-08-16T00:00:00+00:00
Unknown
https://bloerg.net/posts/redesign/<p>You might have noticed a slight visual change of this site. Well, I just pulled
the trigger and re-wrote this site’s CSS based on <a href="http://milligram.io/">milligram</a> in about an
hour. The sole reason for that was that the old CSS was an unmaintable, barely
responsive mess. I am not a CSS expert, so why not leverage one of the countless
minimal CSS frameworks and add a few <a href="https://bloerg.net/static/css/custom.css">customizations</a> to get away from that
hip and modern low-contrast look?</p>
<p>With that came a few changes that made this site leaner and cleaner: I removed
the few <a href="http://fontawesome.io/">Font Awesome</a> icons thus avoiding one more stylesheet and loading of
a web font, changed the heading font from Lato to Roboto Condensed and removed the
Unica One font which was used only in the header bar.</p>
<p>All in all I am pretty happy with the change and the fresh look.</p>
Use meson2017-07-14T00:00:00+00:002017-07-14T00:00:00+00:00
Unknown
https://bloerg.net/posts/use-meson/<p>For the first time in a long time I felt happy about a piece of software: it’s
<a href="http://mesonbuild.com/">meson</a> the build system. I <a href="/2012/11/10/cmake-and-distutils.html">wrote</a> <a href="/2013/10/29/turning-cmake-into-a-good-unix-citizen.html">about</a> <a href="/2015/03/06/pkg-config-variables-in-cmake.html">my</a>
adventures with CMake but in the end I was never really happy with it. Be it its
arcane syntax, intransparent build process or lack of following standards common
in the nix world. I used it but grudgingly because I could not let my colleagues
be burdened with Autotools.</p>
<p>I read about meson about the time it was first announced but did not dare to
dive into it because of lack of maturity and long-term perspective. However,
learning that a lot of GNOME and systemd related software is moving from
Autootools to meson, I changed my mind and read the great <a href="http://mesonbuild.com/Manual.html">manual</a> in order to
convert some of our bigger C projects over from CMake. And what a joy it was!</p>
<p>First the build language: it’s clear, simple and deliberately not
Turing-complete. Unlike what a lot of folks imply when they hear that meson is
written in Python, the language itself is a separate domain specific language.
Many features of this language do not feel like an afterthought but very helpful
to describe build artifacts and build related information. For example string
and list processing such as</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>foo = ' '.join([bar, 'baz'])
</span></code></pre>
<p>feels a lot more robust than CMake’s</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>SET(foo "${bar} baz")
</span></code></pre>
<p>Having a real boolean type reduces the amount of guesswork in situations
with conditionals considerably. Having a Turing-incomplete language with an
object-oriented approach seems verbose, I had the reverse feeling that I was
writing less but more explicit code. A good thing in my opinion.</p>
<p>The best part on Linux is that meson does not try to depart too much from the
conventional <code>./configure && make && make install</code> dance. <code>meson</code> works exactly
like <code>configure</code> except for the fact that it enforces out-of-source builds. In
this step, meson uses <code>pkgconfig</code> in a first class way to find a library’s
compile and link flags unlike CMake which for years suggested to use the
included modules. In the other direction, meson has first-class support for
<code>pkgconfig</code> file generation, with CMake having you to abuse <code>configure_file()</code>
and still get it wrong. There other tool-specific modules such as the <code>gnome</code>
module that let you finish your job in a few lines instead of constructing new
build targets with fragile command construction.</p>
<p>The configuration times for me with meson are almost ten times shorter than
CMake but this probably varies from project to project. The build times are
equally long if you opt-in to use CMake with <a href="https://ninja-build.org">Ninja</a>,
the default build tool of meson. meson’s build output is a bit nicer because it
does not spam the console if there are no warnings or errors during the build.</p>
<p>For now I found only one thing that would have to let me go back to CMake once
in a while: meson requires Python 3.4 and newer. This is not the case on a few
machines I still have to work on, but time will let these phase out too.</p>
Creating an Inkscape Flatpak2017-02-13T00:00:00+00:002017-02-13T00:00:00+00:00
Unknown
https://bloerg.net/posts/creating-an-inkscape-flatpak/<p>As seen previously, installing a Flatpak is a relatively simple matter – and
with Flatpak bundles even a one-click story. Now, building a Flatpak and the
corresponding repository is a slightly different story that I want to tell you
with my adventure making a Flatpak for the latest 0.92 version of Inkscape.</p>
<p>I won’t go into detail about the build process itself which is described
just alright in the Flatpak developer <a href="http://docs.flatpak.org/en/latest/index.html">documentation</a> and a <a href="https://blogs.gnome.org/alexl/">series of
blogposts</a> by Flatpak developer Alexander Larsson. Unfortunately, it is not
100% complete and I had to resort to the manpages and existing Flatpak
descriptions of other projects to get going.</p>
<p>So first thing you want to do is skip the manual <code>flatpak build</code> process and
use the <code>flatpak-builder</code> tool right away. You want to make a package for a real
program and not some toy that does not have any external dependencies anyway,
right? So that’s what I did: I wrote a <a href="https://github.com/matze/inkscape-flatpak/blob/master/org.inkscape.Inkscape.json">org.inkscape.Inkscape.json</a>, listing each and
every dependency of Inkscape. Now here is a message to the dear runtime
providers: please provide a runtime for applications based on the C++ versions
of the GNOME libraries such as gtkmm. I wasted tons of brain matter and CPU
cycles hunting down and compiling<sup class="footnote-reference"><a href="#1">1</a></sup> all the correct library versions<sup class="footnote-reference"><a href="#2">2</a></sup>.
Speaking of depdendency hell: the dependency “resolution” of <code>flatpak-builder</code>
is anything but smart. It caches build output but marks it dirty if <em>any</em> of the
previously declared modules is changed even if there is no direct dependency
between those two modules. Yay!</p>
<p>But besides those issues making the package, pushing the repo to a server and
installing the package from a remote is actually a pleasant ride …</p>
<figure>
<img src="/images/2017-02-13/inkscape.png" />
</figure>
<p>If I find a cheap location to store the repo and bundle, I will let you know.</p>
<p><em>Update</em>: You can find the resulting Flatpak bundle on the main
<a href="https://inkscape.org/en/download/linux/">inkscape.org</a> website.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Have I told you what a mess Boost is? C++: ✓. Custom build tool: ✓. SourceForge download: ✓.
<sup class="footnote-reference"><a href="#2">2</a></sup>: A salute to our brave package maintainers!</p>
</div>
How to install LibreOffice 5.3 from flatpak on Ubuntu 16.042017-02-02T00:00:00+00:002017-02-02T00:00:00+00:00
Unknown
https://bloerg.net/posts/how-to-install-libre-office-from-flatpak/<p>This is a follow-up to <a href="https://blog.simos.info/how-to-install-libreoffice-5-3-on-ubuntu-16-04-from-snap/">Simos</a> post on how to install LibreOffice but using
Flatpak instead of Snap.
If you haven’t have the flatpak system installed yet you must install it first from
Alex Larsson’s PPA</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo add-apt-repository ppa:alexlarsson/flatpakrepo
</span><span>sudo apt update
</span><span>sudo apt install flatpak
</span></code></pre>
<p>As per the <a href="http://flatpak.org/apps.html">instructions</a> you can then add the GNOME runtime
repository</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>flatpak remote-add --if-not-exists gnome https://sdk.gnome.org/gnome.flatpakrepo
</span></code></pre>
<p>download the Flatpak</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>wget http://download.documentfoundation.org/libreoffice/flatpak/latest/LibreOffice.flatpak
</span></code></pre>
<p>and install it</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>flatpak install --bundle LibreOffice.flatpak
</span></code></pre>
<p>That’s it. Localized and runnable straight from whatever launcher you are
using:</p>
<figure>
<img src="/images/2017-02-02/libreoffice.png" />
</figure>
RTFM2016-08-22T00:00:00+00:002016-08-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/rtfm/<p>Yeah, I should have read that f’n
<a href="https://help.gnome.org/users/gnome-terminal/stable/">manual</a> of GNOME’s
standard terminal and I would not have to get creative to select a block of text
from some program’s output<sup class="footnote-reference"><a href="#1">1</a></sup>. Now, I feel embarrassed for not having pressed
<kbd>Ctrl</kbd> while dragging the mouse all these years.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>My go-to solution usually was to pipe the output into a text file and use Vi’s <kbd>Ctrl</kbd>+<kbd>V</kbd> command.</p>
</div>
Upgrade stories2016-08-04T00:00:00+00:002016-08-04T00:00:00+00:00
Unknown
https://bloerg.net/posts/upgrade-stories/<p>I just finished upgrading this machine from Ubuntu 14.04.5 to 16.04.1. I always
have a slight uneasy feeling upgrading a machine remotely but – knock on wood
– it has been a pretty straightforward process even though I had a minor boot
issue which I did not had when going from 12.04 to 14.04.</p>
<p>Besides the nginx web server and postfix mail server, I run <a href="https://gogs.io">Gogs</a>,
<a href="/2015/11/18/syncing-things.html">Syncthing</a> and my commenting system. To start the latter three I used to use
<a href="http://supervisord.org/">Supervisor</a> but again I had the uneasy feeling that it is too hackish and
resource intensive on that small VPS for 24/7 operation<sup class="footnote-reference"><a href="#1">1</a></sup>. Now that Ubuntu
moved from Upstart to systemd, I converted the Supervisor configurations to
systemd <code>.service</code> files which is straightforward because both file formats are
pretty similar. I added those to my <a href="https://www.ansible.com">Ansible</a> playbook and here we go,
everything integrates nicely and smoothly.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>I have to admit, I did not experience any problems at all though but the Python dependency is just <em>meh</em>.</p>
</div>
A sign of life2016-08-02T00:00:00+00:002016-08-02T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-sign-of-life/<p>Before you wonder: yes, I am still alive and I just reviewed all pending
comments. Sorry, for letting you wait so long but in this part of the world, it
was summer.</p>
<p>Anyway, I will resume blogging but will shift topics slightly. For once, I
started investigating <a href="https://www.rust-lang.org">Rust</a>. I implemented <a href="https://en.wikipedia.org/wiki/Netcat">netcat</a> for fun, which took
half an hour to implement and two hours to make alright, i.e. replace all those
<code>.unwrap()</code>s with proper error handling. In fact, I had to rewrite that little
program three times in order to have a somewhat sane structure. Besides, Rust, I
was pretty involved with playing bass in different groups. You won’t see me
writing about them but I probably will write a bit about transcriptions and
theory. That’s about that.</p>
Ledger import of Deutsche Bank data2016-02-22T00:00:00+00:002016-02-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/ledger-import-of-deutsche-bank-data/<p><em>Disclaimer:</em> The main idea of this post might only be interesting for ten other
people in the entire world but the general notion of keeping a double-entry
booking journal might also appeal to you. Note though that I am not a finance
expert, so your mileage may vary.</p>
<h3 id="double-entry-bookkeeping-with-ledger">Double-entry bookkeeping with ledger</h3>
<p>For a very long time, I have not taken particular interest in my finances and my
main strategy has always been spending less than what I earned. This has worked
quite well, however, with regular income and a large variety of expenses such as
insurances, retirement funds, daily things etc., I got overwhelmed a bit and
felt losing control. To get rid of this uneasy feeling, I had to record whatever
I am gaining or spending. As a layperson one has two options to record this:
keep running totals in a spreadsheet – also known as single-entry bookkeeping
– or use a double-entry bookkeeping software to model the flow of money between
different accounts representing income, expenses, assets, liabilities and
equities. Although the former is probably easier to grasp at the beginning, you
will run into limitations such as difficult error detection soon enough. As a
CLI person, I was thus soon attracted to the ledger-like family of double-entry
bookkeeping systems.</p>
<p>The idea of [ledger][] is simple: you record transactions between accounts in a
simple journal text file and then let ledger report balances. Unlike GUI
programs such as GnuCash, ledger never touches your data. This separation of
concerns eases editing (e.g. Vim), revision control (e.g. Git) and greatly
improves interoperability. The latter is also the main reason for the many
different ledger clones of which I want to highlight [hledger][]. Contrary to
ledger that is written in C++, hledger is based on Haskell and is in my
experience a lot more stable. To get more information about accounting with
ledger-likes, please refer to the very extensive documentation of both ledger
and hledger.</p>
<h3 id="data-import-with-hledger">Data import with hledger</h3>
<p>Now, here’s the actual piece covering the title of this post. I have a checking
account with Deutsche Bank<sup class="footnote-reference"><a href="#1">1</a></sup> that provides export of all transactions in CSV
format. As with probably all corporate data formats, the CSV transaction format
has many quirks and pecularities that need to be fixed before appending
everything to a ledger journal. I used to use a homegrown Python script to fix
up the CSV format and output ledger format, however hledger provides a fantastic
<code>csv</code> command that already does the last step perfectly fine. So if you are in
the same boat and want to import Deutsche Bank data into your ledger journal
here is what you need to do:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#5597d6;">tail -n</span><span> +6 $</span><span style="color:#5597d6;">1 </span><span style="color:#72ab00;">| </span><span style="color:#5597d6;">head -n -1 </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">iconv -f</span><span> ISO8859-1</span><span style="color:#5597d6;"> -t</span><span> utf8 </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/([0-9]+)\.([0-9]+),([0-9]+)/\1\2\.\3/' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/([0-9]+),([0-9]+)/\1\.\2/' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/"//g' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/([A-Z][a-z]+), ([A-Z][a-z]+)/\2 \1/' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/,/./g' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/;/,/g' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">hledger -f</span><span> -</span><span style="color:#5597d6;"> --rules-file</span><span style="color:#72ab00;">=</span><span>db.csv.rules print </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'/^$/d' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/(.*) ; (.*)/\1\n ; \2/' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/^(20[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9]) (.*)/\1 \2/' </span><span style="color:#72ab00;">| </span><span>\
</span><span style="color:#5597d6;">sed -E </span><span style="color:#d07711;">'s/ *$//'
</span></code></pre>
<p>Let’s go through this line-by-line in case you want to add your own adaptions.
First we skip the funny six line header and remove the totaling sum at the
bottom. Then we need to convert the encoding from ISO 8859 to UTF-8, otherwise
hledger would give up reading the data. In the next lines, we convert the amount
format from German 1234.00,50 EUR to 1234.50 EUR, remove quotes, replace “Doe,
John” with “John Doe”, commas with dots and semicolons with commas. Then we push
the cleaned data to hledger which uses a <code>db.csv.rules</code> file to interpret the
CSV file and determine account transformations based on some heuristics. Here
is the shortened <code>db.csv.rules</code> file that should get you running:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>account1 Assets:Checking
</span><span>account2 ReplaceMe
</span><span>
</span><span>fields date,,,description,type,,,,,,,,,,out,in,
</span><span>date-format %d.%m.%Y
</span><span>
</span><span>comment %type
</span><span>amount %in%out EUR
</span><span>
</span><span># ... patterns to determine accounts
</span></code></pre>
<p>Because I like to specify amounts as “1234 EUR” rather than “EUR1234”, I have to
construct them manually with the <code>amount</code> statement. After letting hledger do
its job the shell script takes over for final post-processing, including removal
of empty lines between transactions<sup class="footnote-reference"><a href="#2">2</a></sup>, date formatting and trailing white
space removal. Quite a bit of upfront work, but tremendously helpful for
day-to-day operations.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Yes, yes, I know …
<sup class="footnote-reference"><a href="#2">2</a></sup>: They just look plain wrong with vim-ledger’s collapsed folds.
[ledger]: http://ledger-cli.org/
[hledger]: http://hledger.org</p>
</div>
Modern Beamer theme revisited2015-12-10T00:00:00+00:002015-12-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/modern-beamer-theme-revisited/<p>According to the stars and forks<sup class="footnote-reference"><a href="#1">1</a></sup>, the <a href="https://github.com/matze/mtheme">Beamer theme</a> that I [introduced][]
a year ago has filled a longstanding desire for a theme that has a simple and
clean default look. It started as a one man effort but over the year I received
a lot of contributions from many people, first and foremost from [Benjamin
Weiss][] – author of the [HSRM theme][] that was a huge inspiration for this
theme – and [Ross Churchley][]. With their help, the theme transformed from a
stupid hack into a polished and highly customizable Beamer theme called
<em>Metropolis</em> that is now available on [CTAN][]<sup class="footnote-reference"><a href="#2">2</a></sup>. In this post, I will quickly
go through the most important changes.</p>
<h3 id="customization">Customization</h3>
<p>Many aspects of the layout that were formerly hardcoded can now be modified
using pgfkey-based package options. For example, to change the per-frame
progress bar and disable the frame numbering, you can simply pass the following
options when loading the theme</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\usetheme</span><span>[
</span><span> outer/progressbar=foot,
</span><span> outer/numbering=none
</span><span>]{metropolis}
</span></code></pre>
<figure>
<img src="/images/2015-12-10/default.png" />
</figure>
<p>The color theme is also normalized and derives the entire look from only a few
color specifications. For instance, to modify the general look you would change
the normal text and alerted text colors:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\definecolor</span><span>{Purple}{HTML}{911146}
</span><span style="color:#b39f04;">\definecolor</span><span>{Orange}{HTML}{CF4A30}
</span><span>
</span><span style="color:#7f8989;">% Theme colors are derived from these two elements
</span><span style="color:#b39f04;">\setbeamercolor</span><span>{alerted text}{fg=Orange}
</span><span>
</span><span style="color:#7f8989;">% ... however you can of course override styles of all elements
</span><span style="color:#b39f04;">\setbeamercolor</span><span>{frametitle}{bg=Purple}
</span></code></pre>
<figure>
<img src="/images/2015-12-10/colors.png" />
</figure>
<p>Although, Metropolis is mostly recognized as a <em>Fira Sans</em> theme, the hard
dependency quickly became a major pain point for some folks unwilling or unable
to install <em>Fira Sans</em>. With the current version it is finally possible to
compile a theme with PdfTeX or override the font as usual with</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\setsansfont</span><span>[BoldFont={Source Sans Pro Semibold},
</span><span> Numbers={OldStyle}]{Source Sans Pro}
</span></code></pre>
<figure>
<img src="/images/2015-12-10/fonts.png" />
</figure>
<p>You may have noticed that the frame title does not sport small caps by default
anymore. We disabled them because there were too many problems lowercasing the
title and typesetting the title with <code>\textsc{}</code>. However, if you are like me
and prefer the small caps styling, you can get them back easily with</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\metroset</span><span>{titleformat=allsmallcaps}
</span></code></pre>
<figure>
<img src="/images/2015-12-10/allsmallcaps.png" />
</figure>
<p>As you can see, <code>\metroset{}</code> can be used within the document to change options
on the fly. Moreover, there is a second option for a semi small caps style,
i.e. all uppercased letters have a slightly larger x-height:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\metroset</span><span>{titleformat=smallcaps}
</span></code></pre>
<figure>
<img src="/images/2015-12-10/smallcaps.png" />
</figure>
<h3 id="documentation">Documentation</h3>
<p>Originally, the entire theme was documented using the top-level README. With a growing
number of options this approach became a bit tedious and the README totally
overcrowded. Moreover, distribution on CTAN and the likes recommends a proper
manual in PDF format. Thus, we moved all documentation to a real [manual][] and
limit the README to installation and basic usage information. The manual itself
is now derived from the documented TeX source.</p>
<h3 id="the-future">The future</h3>
<p>This is the first version in a series of stable releases, so you can expect a
better theme – but still adhering to its core principles – soon. Nevertheless,
the theme lives from outside contributions, be it code or bug reports, so do not
hesitate to contact us on GitHub.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>As of now the <a href="https://github.com/matze/mtheme">repository</a> has been starred more than 800 times and cloned by more than 150 people.
<sup class="footnote-reference"><a href="#2">2</a></sup>: … and TeXlive and MikTeX.
[introduced]: /2014/09/20/a-modern-beamer-theme.html
[CTAN]: http://www.ctan.org/pkg/beamertheme-metropolis
[Benjamin Weiss]: https://github.com/benjamin-weiss
[HSRM theme]: https://github.com/benjamin-weiss/hsrmbeamertheme
[Ross Churchley]: http://rosschurchley.com
[manual]: http://mirrors.ctan.org/macros/latex/contrib/beamer-contrib/themes/metropolis/doc/metropolistheme.pdf</p>
</div>
Updates in Vimland 20152015-12-08T00:00:00+00:002015-12-08T00:00:00+00:00
Unknown
https://bloerg.net/posts/updates-in-vimland-2015/<p>It’s time for the forth installment of updates concerning my Vim configuration.
Unlike previous years, I made a lot less changes to my <code>.vimrc</code> over the year.
Although, my Vim configuration converges towards a stable setup, I am still not
one hundred percent satisfied with. My biggest gripe is a change in NeoComplete
that sometimes completes gibberish at inappropriate times, for example when I
hit the return key for a new line, NeoComplete it completes whatever it
suggested on the previous line.</p>
<p>You will be surprised to hear that I sticked with Plug as my go-to plugin
manager. There is no new one on the scene or at least none that would make any
difference. However, I started to lazy-load all plugins as much as possible in
order to improve startup latencies.</p>
<p>The behind-the-scences plugin highlight of the year is <a href="https://github.com/ludovicchabant/vim-gutentags">Gutentags</a>. It’s a
non-obtrusive way to generate tags for a project. Before that I updated the
Ctags file with a keymap, however, that obviously causes out-of-sync problems
from time to time. But these times are long forgotten.</p>
<p>Concerning newly acquired builtin Vim mechanics: I fully embraced the
<kbd>f</kbd>, <kbd>t</kbd>, <kbd>;</kbd> and <kbd>,</kbd> keys that I <a href="/2015/08/13/sneaky-vim-motions.html">wrote</a>
about, resulting in huge productivity gains!</p>
<p>That’s it for this year. If you have any Vim secret sauce: let me know!</p>
Syncing things2015-11-18T00:00:00+00:002015-11-18T00:00:00+00:00
Unknown
https://bloerg.net/posts/syncing-things/<p>Even though I could <a href="/2014/04/28/writing-a-git-annex-plugin-for-ranger.html">reduce</a> the pain of using git-annex on the command line
with a Ranger plugin, the weird concept and the workflow concerning files which
are not read-only let me use it less and less. Moreover, when collaborating with
others, I could feel that Dropbox does something right with their system which
was not possible to with git-annex in any way. However, since Dropbox is a
US-based, commercial system, I was looking for alternatives …</p>
<h2 id="peer-to-peer-synchronization-with-syncthing">Peer-to-peer synchronization with Syncthing</h2>
<p><a href="https://syncthing.net">Syncthing</a> is a dead simple tool to organize arbitrary synchronization
networks between devices. Install the <code>syncthing</code> binary and point your browser
to <a href="http://localhost:8384">localhost:8384</a>. Each device that you run Syncthing
on gets a device ID – a secure certificate really – for identification.
Devices who know each others device IDs form your personal cluster. You can then
setup shared <em>folders</em> between subsets of that cluster that are then
synchronized securely using TLS.</p>
<figure>
<img src="/images/2015-11-18/syncthing.png" />
</figure>
<p>As you can see, I run Syncthing on three different machines, one is currently
off, while the one I currently configure and another are able to synchronize.
Note, though that Titan and ipemv1 cannot reliably see each other<sup class="footnote-reference"><a href="#1">1</a></sup>, I use an
intermediate device to distribute data from these four folders. This device’s
rescan interval is set to a relatively large value because I usually won’t
change data <em>from</em> there.</p>
<h2 id="running-syncthing-on-a-vps">Running Syncthing on a VPS</h2>
<p>To have a real Dropbox replacement you will need at least one server that is
always on – for example a cheap VPS – which will act as the intermediate file
server. Be aware though that in case you want to configure this server via the
internet, you will either have to secure the connection with HTTPS<sup class="footnote-reference"><a href="#2">2</a></sup> and set up
a user and a password or tunnel the HTTP port via SSH to your local machine.</p>
<p>To enable HTTPS, you replace the <code>https-cert.pem</code> and <code>https-key.pem</code> in
<code>~/.config/syncthing</code> or use the default created by Syncthing and modify</p>
<pre data-lang="xml" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-xml "><code class="language-xml" data-lang="xml"><span style="color:#6486ab;"><gui enabled=</span><span style="color:#d07711;">"true" </span><span style="color:#6486ab;">tls=</span><span style="color:#d07711;">"false"</span><span style="color:#6486ab;">>
</span></code></pre>
<p>in <code>~/.config/syncthing/config.xml</code> accordingly. To enable SSH tunneling, just
forward the remote port (in this case 8384) to some local port which you then
use to access the configuration page:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ ssh -L 9876:127.0.0.1:8384 whereever
</span><span>$ firefox http://localhost:9876
</span></code></pre>
<p>Ignoring these measures could seriously harm your setup!</p>
<h2 id="long-term-use">Long-term use</h2>
<p>Because I only set up this system recently, I do not have any long-term
experiences to share with you. Most importantly, I have no clue how often and
severe conflicts happen or if the [delta sync algorithm][] is good enough for my
use cases. Only time will tell, but so far I am more than happy to retire
git-annex for a much simpler setup.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>One is behind the home router, the other only accessible via VPN. Moreover, either one is off while the other is on.
<sup class="footnote-reference"><a href="#2">2</a></sup>: One more good reason to get a [free certificate][].
[free certificate]: /2015/11/05/now-served-with-https.html
[delta sync algorithm]: https://forum.syncthing.net/t/better-delta-sync-algorithm-like-rsync/984</p>
</div>
Noerdbier launch2015-11-12T00:00:00+00:002015-11-12T00:00:00+00:00
Unknown
https://bloerg.net/posts/noerdbier-launch/<p>Craft beer and home brewing is a hip topic for quite some time already. For
almost three years now, two friends of mine and I have been brewing various
top-fermented beers. We started with small batches made entirely by hand and
went up to semi-automatic mashing and better controlled equipment<sup class="footnote-reference"><a href="#1">1</a></sup>.</p>
<p>Although we <em>tried</em> to document the entire brewing and fermenting process from
the get-go, we were always lacking the discipline to keep a good journal.
Moreover, we did not have a place to store this in an accessible way for later
review of the final products. In the long run, this would have helped improving
our recipes over time. Better late than never, Max sat down and made this thing
a reality: our Jekyll-based <a href="http://noerdbier.de">Noerdbier</a> home. Right now,
you can find our current batch of a dry-hoped
<a href="http://noerdbier.de/sude/2015/09/19/india-pale-ale/">IPA</a>, a regular
<a href="http://noerdbier.de/sude/2015/09/19/extra-milk-stout/">stout</a> and a chocolatey
<a href="http://noerdbier.de/sude/2015/09/19/extra-chocolate-stout/">modification</a> of
that stout. So, if you speak German, head over to
<a href="http://noerdbier.de">noerdbier.de</a>.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>You can find our open source control software at <a href="https://github.com/brewpeople">GitHub</a>.</p>
</div>
Now served with HTTPS2015-11-05T00:00:00+00:002015-11-05T00:00:00+00:00
Unknown
https://bloerg.net/posts/now-served-with-https/<p>Thanks to the generous offerings from the <a href="https://letsencrypt.org/">Let’s Encrypt</a> initiative and
certificate authority, you can now find all content securely <a href="https://bloerg.net">hosted</a>. Because
I am still somewhat suspicious of the automatic certification process provided
by the official <a href="https://github.com/letsencrypt/letsencrypt">client</a> and I serve this site with Nginx, I followed along
the description of <a href="https://github.com/diafygi/letsencrypt-nosudo/">letsencrypt-nosudo</a> for a semi-automatic process. All in
all, it took about 15 minutes to get a free SSL certificate recognized by
Firefox and Chrome.</p>
<figure>
<img src="/images/2015-11-05/bloerg-ssl.png" />
</figure>
<p><em>Update</em>: looks like one can also <a href="https://github.com/letsencrypt/letsencrypt/pull/502">sign</a> semi-automatically using the
official client.</p>
<p><em>Update</em>: Any problems should be solved now with the Let’s Encrypt certificate
signed by IdenTrust.</p>
Interfacing OpenCL with PCIe devices2015-09-17T00:00:00+00:002015-09-17T00:00:00+00:00
Unknown
https://bloerg.net/posts/interfacing-opencl-with-pcie-devices/<p>Pushing data from a PCIe device to a GPU or letting a GPU write into a PCIe
device directly is necessary to optimize for latencies by avoiding an
intermediate hop through the system memory. For this purpose NVIDIA provides the
GPUDirect technology which is CUDA only. Your only bet to achieve the same on
OpenCL is to buy AMD cards and use the open DirectGMA (marketing speak) or
bus-addressable memory (technical speak) OpenCL extension. However, the
<a href="https://www.khronos.org/registry/cl/extensions/amd/cl_amd_bus_addressable_memory.txt">documentation</a>
is both sparse and out-of-date, which is why I want to give a quick but complete
overview how to transfer data back and forth between a PCIe device and a GPU.</p>
<h2 id="preliminaries">Preliminaries</h2>
<p>Obviously, you will need the AMD OpenCL SDK in order to use the AMD extension,
so get that and set up your build environment accordingly. Besides, the normal
<code><CL/cl.h></code> include you have to include <code><CL/cl_ext.h></code>. This extension allows
data transfers in both directions, however only a pusing is possible in either
direction. That means, when transfering from data the device to the GPU, the
device <em>writes</em> to the GPU and vice versa.</p>
<p>To call functions exported by an extension, you have to load it via an OpenCL
mechanism because of the way OpenCL function calls are dispatched through the
ICD loader. For example, to call <code>clEnqueueMakeBuffersResidentAMD</code> you would
load it beforehand like that:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span>clEnqueueMakeBuffersResidentAMD_fn clEnqueueMakeBuffersResidentAMD;
</span><span>
</span><span>clEnqueueMakeBuffersResidentAMD </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">clGetExtensionFunctionAddressForPlatform </span><span>(platform,
</span><span> </span><span style="color:#d07711;">"clEnqueueMakeBuffersResidentAMD"</span><span>);
</span><span>
</span><span style="color:#72ab00;">if </span><span>(clEnqueueMakeBuffersResidentAMD </span><span style="color:#72ab00;">== </span><span style="color:#b3933a;">NULL</span><span>) {
</span><span> </span><span style="color:#b39f04;">fprintf </span><span>(stderr, </span><span style="color:#d07711;">"Could not load clEnqueueMakeBuffersResidentAMD"</span><span>);
</span><span>}
</span></code></pre>
<p>For brevity reasons, I omit error checking in the remaining post, but you should
<em>always</em> check <em>every</em> possible error.</p>
<h2 id="writing-to-the-gpu">Writing to the GPU</h2>
<p>To write from a PCIe device to the GPU you have to let the PCIe device know the
bus address of a <code>cl_mem</code> object. For this you create a buffer with the
<code>CL_MEM_BUS_ADDRESSABLE_AMD</code> flag (and any other except for
<code>CL_MEM_ALLOC_HOST_PTR</code> and <code>CL_MEM_USE_HOST_PTR</code>)</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span>cl_mem buffer;
</span><span>cl_mem_flags flags;
</span><span>
</span><span>flags </span><span style="color:#72ab00;">=</span><span> CL_MEM_BUS_ADDRESSABLE_AMD;
</span><span>buffer </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">clCreateBuffer </span><span>(context, flags, size, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#72ab00;">&</span><span>error);
</span></code></pre>
<p>and “pin” it before using it from the remote PCIe device</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span>cl_bus_address_amd addr;
</span><span>
</span><span style="color:#5597d6;">clEnqueueMakeBuffersResidentAMD </span><span>(queue,
</span><span> </span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#7f8989;">/* in this case we pin only one buffer */
</span><span> </span><span style="color:#72ab00;">&</span><span>buffer, </span><span style="color:#7f8989;">/* array of buffers */
</span><span> CL_TRUE, </span><span style="color:#7f8989;">/* block */
</span><span> </span><span style="color:#72ab00;">&</span><span>addr, </span><span style="color:#7f8989;">/* array of bus addresses */
</span><span> </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">NULL </span><span style="color:#7f8989;">/* event infrastructure... */</span><span>);
</span></code></pre>
<p>Making the buffer resident fills the <code>surface_bus_address</code> member of <code>addr</code>
(unlike the extension docs which call it <code>surfbusaddress</code>) that you then pass to
your PCIe device one way or another. Now the device can write in whatever way,
however for full speed a DMA transfer is much preferred.</p>
<p>Note that a bus-addressable buffer usually cannot be as large as reported for
<code>CL_DEVICE_MAX_MEM_ALLOC_SIZE</code> using <code>clGetDeviceInfo</code>. For example, on a
FirePro W9100 buffers with a size of about 96 MB can be allocated at most.</p>
<h2 id="writing-to-the-device">Writing to the device</h2>
<p>The most important thing to figure out is the <em>physical</em> memory address of your
device. This information might come from the device’s driver but a quick look
for memory regions using <code>lspci -vv</code> might give you a hint …</p>
<p>To set up a buffer for writing, we do the reverse operation of writing to the
GPU: we set up a <code>cl_bus_address_amd</code> structure with the bus address of the PCIe
device and pass that and the <code>CL_MEM_EXTERNAL_PHYSICAL_AMD</code> flag to
<code>clCreateBuffer</code>:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span>cl_mem remote_buffer;
</span><span>cl_mem_flags flags;
</span><span>cl_bus_address_amd addr;
</span><span>
</span><span>flags </span><span style="color:#72ab00;">=</span><span> CL_MEM_EXTERNAL_PHYSICAL_AMD;
</span><span>addr.</span><span style="color:#5597d6;">surface_bus_address </span><span style="color:#72ab00;">= </span><span>(cl_ulong) physical_bus_address;
</span><span>addr.</span><span style="color:#5597d6;">marker_bus_address </span><span style="color:#72ab00;">= </span><span>(cl_ulong) physical_bus_address;
</span><span>
</span><span>remote_buffer </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">clCreateBuffer </span><span>(context, flags, size, </span><span style="color:#72ab00;">&</span><span>addr, </span><span style="color:#72ab00;">&</span><span>error);
</span></code></pre>
<p>The surface bus address is the main address that you want to access from within
a kernel or when copying between GPUs. You can now use <code>buffer</code> in regular
kernels as if it were memory on the GPU. Be aware though that the addresses must
be page-aligned however anything else should actually be pretty uncommon.</p>
<p>The marker address is a special memory region that is used for inter-context
synchronization. To denote that a transfer has finished you would enqueue a
write signal command</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span>value</span><span style="color:#72ab00;">++</span><span>;
</span><span style="color:#5597d6;">clEnqueueWriteSignalAMD </span><span>(queue, remote_buffer, value, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#72ab00;">&</span><span>event);
</span></code></pre>
<p>where <code>value</code> is a monotonically increasing number. Another device listening for
change can then wait on the signal with</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#5597d6;">clEnqueueWaitSignalAMD </span><span>(queue, local_buffer, value, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#72ab00;">&</span><span>event);
</span></code></pre>
Sneaky Vim motions2015-08-13T00:00:00+00:002015-08-13T00:00:00+00:00
Unknown
https://bloerg.net/posts/sneaky-vim-motions/<p>Although I am a Vim user for almost ten years now, I still learn something new
every once in a while that will improve my editing to be more efficient or
comfortable. This time – and I have to admit that it’s almost embarrassing – I
quickly grew accustomed to the <code>f</code> and <code>t</code> motions. In essence
you type <kbd>f</kbd> or <kbd>t</kbd> followed by a character to which the
cursor will jump on or right before. Typing <kbd>;</kbd> and <kbd>,</kbd>
repeats the <em>motion</em> forward and backward. Before this enlightenment, I was
trodding through long lines by repeating <kbd>w</kbd> and <kbd>e</kbd> motions
until I reached my final destination.</p>
<p>There are two reasons why I missed these character-based motions for so long.
First of all <kbd>f</kbd> and <kbd>t</kbd> can only reach targets on the
current line whereas programming tasks often involve line <em>and</em> word crossings.
Second of all, the non-existant visualization of the motion destinations make it
really hard for me to make sense of these motions.</p>
<p>The very handy <a href="https://github.com/justinmk/vim-sneak">vim-sneak</a> plugin alleviates both of these problems with the
additional benefit of allowing two-character search motions using the
<kbd>s</kbd> and <kbd>S</kbd> keys thus covering middle ground between <code>f</code>
motions and full-blown <code>/</code> searches. To skip unused targets, vim-sneak provides
the <em>streak</em> mode similar to <a href="https://github.com/easymotion/vim-easymotion">vim-easymotion</a> that allows to reach a target by
typing a third character.</p>
<figure>
<img src="/images/2015-08-13/comparison.png" />
</figure>
<p>In the given example I was searching for “mo” starting on the first character on
line 6. Streak provides shortcuts <code>a</code> and <code>s</code> to avoid having to type
<kbd>;</kbd> two or three times. To enable the streak mode, you have to add the
following to your <code>.vimrc</code>:</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">let </span><span style="color:#5597d6;">g:sneak</span><span>#streak = </span><span style="color:#b3933a;">1
</span></code></pre>
<p>By default, vim-sneak does not interfere with <code>f</code> and <code>t</code> motions thus to
benefit from highlighting and multi-line <code>f</code> and <code>t</code> motions you have to map the
corresponding keys:</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">nmap</span><span> f </span><span style="color:#a2a001;"><Plug></span><span>Sneak_f
</span><span style="color:#b39f04;">nmap</span><span> F </span><span style="color:#a2a001;"><Plug></span><span>Sneak_F
</span><span style="color:#b39f04;">nmap</span><span> t </span><span style="color:#a2a001;"><Plug></span><span>Sneak_t
</span><span style="color:#b39f04;">nmap</span><span> T </span><span style="color:#a2a001;"><Plug></span><span>Sneak_T
</span></code></pre>
<p>By the way, I link the target colors to type and function syntax colors as
follows to get rid of the pesky default pink:</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span>hi link SneakPluginTarget Type
</span><span>hi link SneakPluginScope Function
</span><span>hi link SneakStreakTarget Type
</span><span>hi link SneakStreakMask Function
</span></code></pre>
Interactive Git rebase2015-07-29T00:00:00+00:002015-07-29T00:00:00+00:00
Unknown
https://bloerg.net/posts/interactive-git-rebase/<p>Lately, a <a href="http://chris.beams.io/posts/git-commit/">post</a> on writing good Git commit messages and the subsequent Reddit
<a href="https://www.reddit.com/r/programming/comments/3evbbk/how_to_write_a_git_commit_message/">discussion</a> caught my attention. I fully agree with each and every point made
and in fact alway try to convince colleagues and friends to follow this model.
However, what to do if you are happily coding away, churning out commit after
commit and then end up with a larger number of commits with summaries such as
“Fixes this” or “Forgot to add that”? You use an interactive rebase, a Git
feature that is surprisingly unknown among my fellow peers. </p>
<p>As the name suggests, an interactive rebase is a rebase done in a more intuitive
way. That does not sound like a lot because most people assume a rebase to be
taking commits from one branch and putting them on top of a commit of another
branch. However, with the default workflow an interactive rebase happens on the
<em>same</em> branch. To initiate such a rebase, simply find a suitable range of commits
that you want to <em>remove</em>, <em>edit</em> or <em>combine</em> (e.g. from <code>HEAD</code> down the last
ten commits) and type</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ git rebase -i HEAD~10
</span></code></pre>
<p>Your favorite editor will then be opened with the list of commits in
chronological order. You can <em>remove</em> commits simply by removing the
corresponding line or <em>reordering</em> commits by moving lines up and down.
Concerning the Git commit messages, you can edit a message by replacing <code>pick</code>
with <code>reword</code> (short <code>r</code>) or <code>edit</code> (short <code>e</code>) which also gives you the
opportunity to change the author of a commit. To <em>combine</em> commits, you can
either use <code>squash</code> (short <code>s</code>) which creates a single commit out of all the
marked commits and the one commit leading to the first squash commit. <code>fixup</code>
does the same but will use the commit message of the previous commit. These
tools help to consolidate related commits to one logical commit.</p>
<p>Interactive rebase is a pleasant way of reorganizing the <em>private</em> history. In
case you are using Vim, this can be an even more pleasing experience with a
small <a href="/2013/12/04/a-tiny-vim-plugin-for-interactive-git-rebasing.html">plugin</a> that I wrote some time ago.</p>
Git alias for checking out pull requests2015-05-12T00:00:00+00:002015-05-12T00:00:00+00:00
Unknown
https://bloerg.net/posts/git-alias-for-checking-out-pull-requests/<p>To fetch a pull requests by its ID, I probably googled “github checkout pull
request” a hundred times by now. Here is a simple Git alias which goes into the
<code>.gitconfig</code> to make that a <em>bit</em> easier:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>[alias]
</span><span> checkout-pr = "!f() { git fetch origin pull/$1/head:pr-$1 && git checkout pr-$1; }; f"
</span></code></pre>
<p>Now it’s just a matter of <code>git checkout-pr 42</code> to check the changes of pull
request #42 on my local system.</p>
Installing Neovim2015-03-23T00:00:00+00:002015-03-23T00:00:00+00:00
Unknown
https://bloerg.net/posts/installing-neovim/<p>Quick reminder how to use the alternatives system to call <a href="http://neovim.io">Neovim</a>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo update-alternatives --install /usr/bin/vi vi /usr/local/bin/nvim
</span><span>sudo update-alternatives --install /usr/bin/vim vim /usr/local/bin/nvim
</span><span>sudo update-alternatives --install /usr/bin/editor editor /usr/local/bin/nvim
</span></code></pre>
<p>… and to change it</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo update-alternatives --config vi
</span></code></pre>
pkg-config variables in CMake2015-03-06T00:00:00+00:002015-03-06T00:00:00+00:00
Unknown
https://bloerg.net/posts/pkg-config-variables-in-cmake/<p><a href="http://www.freedesktop.org/wiki/Software/pkg-config/">pkg-config</a> is a freedesktop standard and implementation that helps
developers finding correct compile and link flags of dependencies. Despite its
very low version number, it is the <em>de facto</em> method for determining said flags
on the Linux desktop. An oft-forgotten feature of pkg-config is the ability of a
package to expose arbitrary key-value meta data. This allows dependent
applications to find data paths, plugins to know where to install themselves or
determining tool paths.</p>
<p>Although CMake supports pkg-config for quite some time with its <a href="http://www.cmake.org/cmake/help/v3.0/module/FindPkgConfig.html">FindPkgConfig</a>
module, it only queries for the compile and link information. If you want to get
a variable value you are out of luck. But fear not! Just stick this into a CMake
module</p>
<pre data-lang="cmake" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cmake "><code class="language-cmake" data-lang="cmake"><span style="color:#b39f04;">find_package</span><span>(PkgConfig </span><span style="color:#5597d6;">REQUIRED</span><span>)
</span><span>
</span><span style="color:#b39f04;">function</span><span>(</span><span style="color:#c23f31;">pkg_check_variable </span><span>_pkg _name)
</span><span> </span><span style="color:#b39f04;">string</span><span>(</span><span style="color:#5597d6;">TOUPPER </span><span>${</span><span style="color:#5597d6;">_pkg</span><span>} _pkg_upper)
</span><span> </span><span style="color:#b39f04;">string</span><span>(</span><span style="color:#5597d6;">TOUPPER </span><span>${</span><span style="color:#5597d6;">_name</span><span>} _name_upper)
</span><span> </span><span style="color:#b39f04;">string</span><span>(</span><span style="color:#5597d6;">REPLACE </span><span style="color:#d07711;">"-" "_" </span><span>_pkg_upper ${</span><span style="color:#5597d6;">_pkg_upper</span><span>})
</span><span> </span><span style="color:#b39f04;">string</span><span>(</span><span style="color:#5597d6;">REPLACE </span><span style="color:#d07711;">"-" "_" </span><span>_name_upper ${</span><span style="color:#5597d6;">_name_upper</span><span>})
</span><span> </span><span style="color:#b39f04;">set</span><span>(</span><span style="color:#5597d6;">_output_name </span><span style="color:#d07711;">"${</span><span style="color:#acb3c2;">_pkg_upper</span><span style="color:#d07711;">}_${</span><span style="color:#acb3c2;">_name_upper</span><span style="color:#d07711;">}"</span><span>)
</span><span>
</span><span> </span><span style="color:#b39f04;">execute_process</span><span>(</span><span style="color:#5597d6;">COMMAND </span><span>${</span><span style="color:#5597d6;">PKG_CONFIG_EXECUTABLE</span><span>} --variable=${</span><span style="color:#5597d6;">_name</span><span>} ${</span><span style="color:#5597d6;">_pkg</span><span>}
</span><span> </span><span style="color:#5597d6;">OUTPUT_VARIABLE </span><span>_pkg_result
</span><span> </span><span style="color:#5597d6;">OUTPUT_STRIP_TRAILING_WHITESPACE</span><span>)
</span><span>
</span><span> </span><span style="color:#b39f04;">set</span><span>(</span><span style="color:#d07711;">"${</span><span style="color:#acb3c2;">_output_name</span><span style="color:#d07711;">}" "${</span><span style="color:#acb3c2;">_pkg_result</span><span style="color:#d07711;">}" </span><span style="color:#5597d6;">CACHE </span><span style="color:#668f14;">STRING </span><span style="color:#d07711;">"pkg-config variable ${</span><span style="color:#acb3c2;">_name</span><span style="color:#d07711;">} of ${</span><span style="color:#acb3c2;">_pkg</span><span style="color:#d07711;">}"</span><span>)
</span><span style="color:#b39f04;">endfunction</span><span>()
</span></code></pre>
<p>and you can easily query information like this</p>
<pre data-lang="cmake" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cmake "><code class="language-cmake" data-lang="cmake"><span style="color:#5597d6;">pkg_check_modules</span><span>(</span><span style="color:#5597d6;">GLIB </span><span>glib-2.0)
</span><span style="color:#5597d6;">pkg_check_variable</span><span>(glib-2.0 glib-genmarshal)
</span><span style="color:#b39f04;">message</span><span>(</span><span style="color:#d07711;">"Path: ${</span><span style="color:#acb3c2;">GLIB_2</span><span style="color:#d07711;">.0_GLIB_GENMARSHAL}"</span><span>)
</span></code></pre>
Comments enabled2015-01-22T00:00:00+00:002015-01-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/comments-re-enabled/<p>I found the time to update my static comment script and will have the time to
review comments again for the foreseeable future. Thus spam bots: ready, steady,
go!</p>
Centralized encrypted file storage2015-01-20T00:00:00+00:002015-01-20T00:00:00+00:00
Unknown
https://bloerg.net/posts/centralized-encrypted-file-storage/<p>In this day and age, cloud storage is a convenient way to access data from any
location. However for privacy and data theft reasons, using third party products
always leaves a sour taste in my opinion. To host my own centralized and
encrypted data storage for small, highly confidential data, I set up a system
combining <a href="http://en.wikipedia.org/wiki/SSHFS">SSHFS</a> and <a href="http://en.wikipedia.org/wiki/EncFS">EncFS</a>.</p>
<p>SSHFS is a FUSE file system to mount a remote file system securely through an
SSH tunnel. As a nice side effect, this means that no additional client setup is
required once the keys are distributed accordingly. EncFS is another FUSE file
system that transparently decrypts an encrypted data source as a mounted
directory. In the remaining post, I will explain how I use both systems together
to read and write securely to an encrypted remote file system on my VPS.</p>
<p>Before mounting the file systems, we need two empty mount points: one that
contains the SSHFS mounted <em>encrypted</em> directory (let’s call the path
<code>ENCBOX_PATH</code>) and one that contains the decrypted data (let’s call that
<code>BOX_PATH</code>). Before mounting the remote file system you should make sure that
the remote directory actually exists. If that’s the case, you mount the remote
directory with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sshfs foo@bar.com:/path/to/box $ENCBOX_PATH -o uid=$(id -u) -o gid=$(id -g)
</span></code></pre>
<p>I pass the <code>uid</code> and <code>gid</code> parameters to map my local user to the remote one.
Otherwise I would not be able to access the data. Once <code>$ENCBOX_PATH</code> is
available we can encrypt the contents with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>encfs $ENCBOX_PATH $BOX_PATH
</span></code></pre>
<p>To avoid having to type in the password every time, we have to pass the password
programmatically to EncFS. You could either get it from somewhere and pass it
through stdin using the <code>--stdin</code> option or specify an external programm that
prints the password on stdout. I use the latter together with my
<a href="https://github.com/matze/configs/blob/master/.local/bin/keyring-query">keyring-query</a> script. It is a small Python script that uses the system’s
keyring to store and retrieve arbitrary secrets. Thus before trying to mount
the encrypted file system I will first store the password using</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>keyring-query --service=encbox --user=foo
</span></code></pre>
<p>The same command line is then used for the <code>encfs</code> command</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>encfs --extpass="keyring-query --service=encbox --user=foo" $ENCBOX_PATH $BOX_PATH
</span></code></pre>
<p>To avoid letting the box stay open indefinitely, I also set EncFS’ <code>--idle</code>
option to five minutes.</p>
<p>Because all those steps are quite elaborate, I bundled this together in a small
<a href="https://github.com/matze/configs/blob/master/.local/bin/boxmount">shell script</a>. Besides the basic mounting of the file systems the script also
takes care of checking error conditions and unmounting the file systems if they
are already mounted.</p>
<p>The outlined solution is a huge relief for keeping confidential data centrally
located that I don’t want to share with Google or Dropbox. However, this comes
at a price: Although the bandwidth is not too bad (around 1.8 MB/s on a fast
line), writing interactively (e.g. editing with Vim) becomes a pain due to the
high latencies involved.</p>
A better LaTeX driver2014-10-31T00:00:00+00:002014-10-31T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-better-latex-driver/<p>Compiling a LaTeX/BibTeX document is a tedious process does not integrate very
well with the <code>make</code> idiom. Until now, I used [rubber][] to compile my
documents. However, there are several shortcomings that make rubber only a
<em>good</em> rather than a <em>perfect</em> solution. Most notably, it is basically
unmaintained for years now with the current release dating back to 2006. This
wouldn’t be a big issue if it weren’t for some annoyances such as in-source
specification of the LaTeX compiler<sup class="footnote-reference"><a href="#1">1</a></sup> and the inability to specify additional
flags (e.g. <code>-shell-escape</code>).</p>
<p>Thanks to Austin Clements and his a spiritual successor to rubber, [latexrun][],
the days of frustration and despair are over now. The name might not be as sexy
but the feature list and improvements are certainly impressive. Most
importantly, latexrun spits out errors and warnings in a format that makes Vim’s
quickfix window happy and I can finally specify commands and arguments
willy-nilly from <em>outside</em> of my document.</p>
<p>(Un)fortunately, latexrun likes to output auxiliary files in a separate build
directory called <code>latex.out</code>. While that works for most packages, it [breaks][]
with the source highlighting package <em>minted</em>. As a temporary workaround, one
can use the <code>-O .</code> option to use the current working directory as the build
directory. But apart from that issue, I am very happy with this little program.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Such as XeLaTeX which I use almost exclusively these days.
[rubber]: /2013/06/25/use-rubber-to-compile-tex.html
[latexrun]: https://github.com/aclements/latexrun
[breaks]: https://github.com/aclements/latexrun/issues/13</p>
</div>
Books and Jekyll rating stars2014-10-29T00:00:00+00:002014-10-29T00:00:00+00:00
Unknown
https://bloerg.net/posts/books-and-jekyll-rating-stars/<p>When a friend found out that I am “blogging”, she assumed I would be posting
personal details and share my views about this and that. Aside from my very
personal configurations however, that is obviously not the case. I’d like to
change that a tiny little bit by linking to a <a href="/books.html">page with a list of
books</a> – see also the header bar – that I read in the past.
This will not only give you a limited idea about myself but also helps me
keeping track of what I read.</p>
<p>To keep this post a bit technical, here is the bit of code that uses <a href="http://fortawesome.github.io/Font-Awesome/">Font
Awesome</a> to display the Jekyll star ratings next to some of the books (props
to <a href="http://techtinkering.com/2011/02/15/a-jekyll-plugin-to-display-ratings-as-star-images/">Lawrence Woodman</a> about the initial idea):</p>
<pre data-lang="ruby" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#72ab00;">module </span><span style="color:#c23f31;">Jekyll
</span><span> </span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">RatingStars </span><span>< </span><span style="color:#a2a001;">Liquid</span><span style="font-style:italic;color:#b06936;">::Tag
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">initialize</span><span>(</span><span style="color:#5597d6;">tag_name</span><span>, </span><span style="color:#5597d6;">text</span><span>, </span><span style="color:#5597d6;">tokens</span><span>)
</span><span> </span><span style="color:#72ab00;">super
</span><span> </span><span style="color:#5597d6;">@stars </span><span style="color:#72ab00;">=</span><span> text
</span><span> </span><span style="color:#72ab00;">end
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">render</span><span>(</span><span style="color:#5597d6;">context</span><span>)
</span><span> rating </span><span style="color:#72ab00;">= </span><span style="color:#b39f04;">Float</span><span>(</span><span style="color:#5597d6;">@stars</span><span>)
</span><span> num_full_stars </span><span style="color:#72ab00;">=</span><span> rating.floor()
</span><span> num_half_stars </span><span style="color:#72ab00;">=</span><span> rating </span><span style="color:#72ab00;">-</span><span> num_full_stars </span><span style="color:#72ab00;">>= </span><span style="color:#b3933a;">0.5 </span><span style="color:#72ab00;">? </span><span style="color:#b3933a;">1 </span><span style="color:#72ab00;">: </span><span style="color:#b3933a;">0
</span><span> num_empty_stars </span><span style="color:#72ab00;">= </span><span style="color:#b3933a;">5 </span><span style="color:#72ab00;">-</span><span> num_full_stars </span><span style="color:#72ab00;">-</span><span> num_half_stars
</span><span>
</span><span> html </span><span style="color:#72ab00;">= </span><span style="color:#b39f04;">String</span><span>.</span><span style="color:#72ab00;">new
</span><span> html </span><span style="color:#72ab00;">+= </span><span style="color:#d07711;">'<span class="rating">'
</span><span>
</span><span> num_full_stars.times </span><span style="color:#72ab00;">do
</span><span> html </span><span style="color:#72ab00;">+= </span><span style="color:#d07711;">'<i class="fa fa-star"></i>'
</span><span> </span><span style="color:#72ab00;">end
</span><span>
</span><span> num_half_stars.times </span><span style="color:#72ab00;">do
</span><span> html </span><span style="color:#72ab00;">+= </span><span style="color:#d07711;">'<i class="fa fa-star-half-o"></i>'
</span><span> </span><span style="color:#72ab00;">end
</span><span>
</span><span> num_empty_stars.times </span><span style="color:#72ab00;">do
</span><span> html </span><span style="color:#72ab00;">+= </span><span style="color:#d07711;">'<i class="fa fa-star-o"></i>'
</span><span> </span><span style="color:#72ab00;">end
</span><span>
</span><span> html </span><span style="color:#72ab00;">+= </span><span style="color:#d07711;">'</span>'
</span><span> html
</span><span> </span><span style="color:#72ab00;">end
</span><span> </span><span style="color:#72ab00;">end
</span><span style="color:#72ab00;">end
</span><span>
</span><span style="color:#a2a001;">Liquid</span><span>::</span><span style="color:#a2a001;">Template</span><span>.register_tag(</span><span style="color:#d07711;">'rating'</span><span>, </span><span style="color:#a2a001;">Jekyll</span><span>::RatingStars)
</span></code></pre>
Updates in Vimland 20142014-10-13T00:00:00+00:002014-10-13T00:00:00+00:00
Unknown
https://bloerg.net/posts/updates-in-vimland-2014/<p>Another year has passed thus it’s time for the third installment of the series
on my Vim configuration. The main motto of this year has been “reduction and
time saving”.</p>
<h2 id="neobundle-becomes-plug-and-airline-becomes-lightline">NeoBundle becomes Plug and Airline becomes Lightline</h2>
<p>I know, I know, yet another plugin manager. This time it’s <a href="https://github.com/junegunn/vim-plug">vim-plug</a>, which has
all the features of NeoBundle has but additionally can install plugins <em>in
parallel</em>. This a huge timesaver when installing or updating some twenty
plugins. This requires <code>+ruby</code> support until we will finally see NeoVim
replacing Bram Moolenaar’s original Vim.</p>
<p>In the process of slimming down my <code>.vimrc</code> and the overall appearance of a
typical Vim session (“less is more”), I replaced Airline with <a href="https://github.com/itchyny/lightline.vim">Lightline</a>. I
removed the majority of additional information except for the color-coded mode,
filename, modified flag and current position. Pure and simple.</p>
<figure>
<img src="/images/2014-10-13/lightline.png" />
</figure>
<h2 id="improved-vim-tex-fold">Improved vim-tex-fold</h2>
<p>I already <a href="/2013/08/22/vim-folding-for-tex.html">wrote</a> about the <a href="https://github.com/matze/vim-tex-fold">plugin</a> that folds parts of a TeX document. Since
the initial release it has seen some major improvements with some input given by
Daniel Andreasen. The plugin should be a bit faster (probably not noticeable)
and more flexible by allowing to fold custom environments. Give it a shot, if
you edit LaTeX documents regularly and do not want to use one of the larger
LaTeX toolboxes.</p>
<h2 id="semi-asynchronous-execution">Semi-asynchronous execution</h2>
<p>Calling out to external programs and libraries in Vim is synchronous and
blocks any user input. This becomes a big problem when dealing with
long-running processes such as compiling program sources or TeX documents with
Make. <a href="https://github.com/neovim/neovim">NeoVim</a>’s main objective is to lift this limitation by replacing
Vim’s synchronous main loop by an event-driven loop. In the mean time
<a href="https://github.com/tpope/vim-dispatch">vim-dispatch</a> fills that role by talking to a surrounding tmux session.
Thus, whenever you run <code>:Make</code> it will start a new instance on the bottom
while allowing you to continue editing the actual text. This has become a
major time saver in the past.</p>
<h2 id="honorary-plugin-mentions">Honorary plugin mentions</h2>
<p>Some more plugins that made it into my <code>.vimrc</code>:</p>
<ul>
<li><a href="https://github.com/junegunn/goyo.vim">Goyo</a> is a <a href="http://www.hogbaysoftware.com/products/writeroom">whiteroom</a>-like plugin that turns off any distracting
elements and adds margins to neatly center the text in the middle. Great for
writing non-code.</li>
<li><a href="https://github.com/tpope/vim-vinegar">vim-vinegar</a> by the venerable Tim Pope is a file opener that complements
CtrlP rather nicely. You type a dash <code>-</code> in normal mode which will take you to
a file list that you can navigate with regular Vi navigation. That comes in
especially handy for files that “neighbour” the current one.</li>
<li><a href="https://github.com/reedes/vim-wordy">vim-wordy</a> uses a huge list of words, phrases and oddities that indicate
poor writing. I only use it occasionally because it tends to nag a bit too
much for my taste. But you should definitely read the background sources which
give a nice overview what to avoid when writing prose.</li>
</ul>
<h2 id="mappings">Mappings</h2>
<p>Probably the biggest change has been remapping the <code><Leader></code> prefix from the
comma key to the space bar. It’s a lot easier now to hit the space bar with my
thumb instead of finding the comma key with my index finger.</p>
<p>Another big timesaver was mapping the command sequence <code>1z=</code> to <code><Leader>ss</code>.
If spell checking is enabled it will replace a wrong word under the cursor by
the first suggestion of the correction list.</p>
A modern Beamer theme2014-09-20T00:00:00+00:002014-09-20T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-modern-beamer-theme/<p>Beamer is an awesome way to make slides with LaTeX. But I always felt that the
<a href="http://deic.uab.es/~iblanes/beamer_gallery/index_by_theme.html">stock</a> themes
do not necessarily look particularly nice and the
<a href="http://latex.simon04.net/">custom</a> themes often scream “Beamer” at first
sight.</p>
<p>So, I sat down and made the <a href="https://github.com/matze/mtheme">mtheme</a> …</p>
<figure>
<img src="/images/2014-09-20/mtheme.png" />
</figure>
<p>In the spirit of <a href="http://en.wikipedia.org/wiki/Jan_Tschichold">Jan Tschichold</a>
and <a href="http://practicaltypography.com/presentations.html">Matthew Butterick</a>, the
theme tries to minimize visual noise and maximize space for content. Thus an
ordinary slide is free of superfluous elements such as a navigation bar,
progress indicators and heavy block elements. What’s left is all the necessary
information: the frame titles, a frame number to refer to the slide and the
actual content.</p>
<h3 id="colors">Colors</h3>
<p>Yes, <a href="http://theabyssgazes.blogspot.de/2010/03/teal-and-orange-hollywood-please-stop.html">teal and
orange</a>
is an annoying fad in movie productions but it is a refreshing choice for a
presentation theme. Rather than using both colors all the time, a dark teal tone
is used for the majority of the Beamer elements while orange is used to
highlight bits and pieces such as <code>\alert{}</code> commands. Instead of going for an
all-white background, I toned it down a tiny notch to reduce eye strain but keep
the contrast high enough to easily discern individual letters.</p>
<h3 id="fonts">Fonts</h3>
<p><a href="https://www.mozilla.org/en-US/styleguide/products/firefox-os/typeface/">Fira Sans</a>
is gorgeous typeface commissioned by Mozilla and designed by
<a href="http://www.carrois.com/fira-3-1/">Carrois</a>. It comes with an enormous amount of
styles and features that cover even the most adventurous presentations. In the
<em>mtheme</em> I used <em>Fira Sans Light</em> as the main font and the regular <em>Fira Sans
Book</em> typeface for any heavier uses such as <code>\textbf{}</code>. <em>Fira Sans Mono</em> is a
perfect match for <code>lstlistings</code> or <code>minted</code> environments.</p>
<p>Titles tend to be difficult to get right because of capitalization which almost
always looks weird. I avoid this problem by lowercasing frame titles and set
them in small capitals of <em>Fira Sans Book</em>.</p>
<h3 id="sections">Sections</h3>
<p>In a presentation it can sometimes be difficult to go from one sub-topic to
another. An elegant solution is to insert a slide whenever a new section starts.
As a little gimmick, the <em>mtheme</em> features a tiny progress bar drawn with TikZ
below the section title.</p>
<h3 id="chart-styles">Chart styles</h3>
<p>A visual representation of numerical data can help to draw conclusions about an
issue quicker than large columns of numbers. Nevertheless, to achieve this
honorable goal, this representation must focus on the <em>data</em> and avoid any
redundant or unnecessary information by increasing the data-ink ratio, a term
coined by Edward Tufte.</p>
<p>I like to use the <a href="http://pgfplots.sourceforge.net/">pgfplots</a> package to
visualize two- and three-dimensional data sets but the default style is –
although better than what Excel offers – not very impressive. To improve the
look, I include the <code>(horizontal) mbarplot</code> and <code>mlineplot</code> styles which reduce
the number of visual elements such as boxes and employ a color scheme that is
(hopefully) easier on the eyes.</p>
<h3 id="conclusion">Conclusion</h3>
<p>Even though this theme might give your presentation a somewhat “professional”
look, you should focus on content first. Because the comments are not back yet,
you can leave any feedback in this
<a href="http://www.reddit.com/r/LaTeX/comments/2gwarq/i_was_dissatisfied_with_the_beamer_themes_so_i/">reddit</a>
thread.</p>
Summer break2014-06-13T00:00:00+00:002014-06-13T00:00:00+00:00
Unknown
https://bloerg.net/posts/summer-break/<p>There haven’t been in any posts for a while and I sincerely have to apologize
that this will continue for at least two months. This will also affect the
comment section, which I will deactivate to reduce the amount of moderation.</p>
Creepy Gmail2014-04-29T00:00:00+00:002014-04-29T00:00:00+00:00
Unknown
https://bloerg.net/posts/creepy-gmail/<p>I knew, Google was scanning my mail to present personalized ads to me but I
never saw them though because of AdblockPlus. I got back to reality, when Gmail
offered me to track the delivery of an Amazon order …</p>
<figure>
<img src="/images/2014-04-29/amazon-gmail.png" />
</figure>
Writing a git-annex plugin for ranger2014-04-28T00:00:00+00:002014-04-28T00:00:00+00:00
Unknown
https://bloerg.net/posts/writing-a-git-annex-plugin-for-ranger/<p>I use git-annex to distribute and synchronize fairly large and mostly static
files across different machines. However, being based on Git makes it pretty
uncomfortable to use from the command line. So, why not integrating it into our
favorite command line file manager,
<a href="/2012/10/17/ranger-file-manager.html">ranger</a>? Because I struggled a bit with
ranger’s internals, I will outline how I wrote <a href="https://github.com/matze/ranger-annex">the
plugin</a>.</p>
<h2 id="initialization">Initialization</h2>
<p>First of all, we need a place for our plugin. By default ranger imports <em>every</em>
Python module from <code>$XDG_CONFIG_HOME/ranger/plugins</code> in lexicographic order. If
<code>$XDG_CONFIG_HOME</code> is not set, <code>~/.config</code> is used as the default alternative.</p>
<p>To use the plugin from within ranger, you need to provide code that hooks into
one of the two methods provided by the ranger API. <code>hook_init</code> is called before
the UI is ready, which means you can dump output on <code>stdout</code>, whereas the UI can
be used in <code>hook_ready</code>. The suggested way is not to replace the original
function but chaining up like this:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">import </span><span>ranger.api
</span><span>
</span><span>old_hook_init </span><span style="color:#72ab00;">= </span><span>ranger.api.hook_init
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">hook_init</span><span>(</span><span style="color:#5597d6;">fm</span><span>):
</span><span> </span><span style="color:#7f8989;"># setup
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#5597d6;">old_hook_init</span><span>(fm)
</span><span>
</span><span>ranger.api.hook_init </span><span style="color:#72ab00;">= </span><span>hook_init
</span></code></pre>
<h2 id="adding-new-commands">Adding new commands</h2>
<p>In the introductory post, I briefly explained how to write custom commands which
you add to your <code>commands.py</code> file: You simply subclass from
<code>ranger.api.commands.Command</code> and write code in the <code>execute</code> method. However,
commands defined in a plugin are not automatically added to the global command
list. For this you need to extend the <code>commands</code> dictionary of the file manager
instance, i.e.</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">hook_init</span><span>(</span><span style="color:#5597d6;">fm</span><span>):
</span><span> fm.commands.commands[</span><span style="color:#d07711;">'annex_copy'</span><span>] </span><span style="color:#72ab00;">= </span><span>copy
</span></code></pre>
<p>When using the <code>annex_copy</code> command, tab-completion should cycle through all available
remotes. This is done by returning an iterable in the <code>tab</code> method. In the
<code>execute</code> method you can access arguments, by calls to the <code>arg</code> method:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">class </span><span style="color:#c23f31;">copy</span><span>(</span><span style="font-style:italic;color:#b06936;">ranger.api.commands.Command</span><span>):
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">tab</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span>(</span><span style="color:#d07711;">'annex_copy </span><span style="color:#aeb52b;">{}</span><span style="color:#d07711;">'</span><span>.</span><span style="color:#5597d6;">format</span><span>(r) </span><span style="color:#72ab00;">for </span><span>r </span><span style="color:#72ab00;">in </span><span>remotes)
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">execute</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> remote </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">arg</span><span>(</span><span style="color:#b3933a;">1</span><span>)
</span></code></pre>
<h3 id="asynchronous-calls">Asynchronous calls</h3>
<p>The git-annex plugin works on the current or currently selected list of files,
which you can get via <code>fm.env.get_selection()</code>. To avoid blocking the UI while
fetching large files, I use the <code>CommandLoader</code> to run the <code>git annex</code> commands
in the background (thanks <a href="https://github.com/hut">@hut</a>). The loader emits a
signal when the action is finished to which we subscribe in order to refresh the
directory content:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">class </span><span style="color:#c23f31;">copy</span><span>(</span><span style="font-style:italic;color:#b06936;">ranger.api.commands.Command</span><span>):
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">execute</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> remote </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">arg</span><span>(</span><span style="color:#b3933a;">1</span><span>)
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">reload_dir</span><span>():
</span><span> </span><span style="color:#5597d6;">self</span><span>.fm.thisdir.</span><span style="color:#5597d6;">unload</span><span>()
</span><span> </span><span style="color:#5597d6;">self</span><span>.fm.thisdir.</span><span style="color:#5597d6;">load_content</span><span>()
</span><span>
</span><span> </span><span style="color:#72ab00;">for </span><span>path </span><span style="color:#72ab00;">in </span><span>(</span><span style="color:#a2a001;">str</span><span>(p) </span><span style="color:#72ab00;">for </span><span>p </span><span style="color:#72ab00;">in </span><span style="color:#5597d6;">self</span><span>.fm.env.</span><span style="color:#5597d6;">get_selection</span><span>()):
</span><span> fname </span><span style="color:#72ab00;">= </span><span>os.path.</span><span style="color:#5597d6;">basename</span><span>(path)
</span><span> cmd </span><span style="color:#72ab00;">= </span><span>[</span><span style="color:#d07711;">'git'</span><span>, </span><span style="color:#d07711;">'annex'</span><span>, </span><span style="color:#d07711;">'copy'</span><span>, </span><span style="color:#d07711;">'-t'</span><span>, remote, fname]
</span><span> loader </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">CommandLoader</span><span>(cmd, </span><span style="color:#d07711;">"annex_copy to remote"</span><span>)
</span><span> loader.</span><span style="color:#5597d6;">signal_bind</span><span>(</span><span style="color:#d07711;">'after'</span><span>, reload_dir)
</span><span> fm.loader.</span><span style="color:#5597d6;">add</span><span>(loader)
</span></code></pre>
<p>Long running actions can be cancelled by the user, so if you need to clean up
you should add that extra code to the <code>cancel</code> method.</p>
<h2 id="wrap-up">Wrap up</h2>
<p>That’s it, a plugin that registers new commands for interacting with git-annex
an asynchronous way. Bug reports and pull requests are welcome.</p>
Re: Advanced vim tips2014-04-18T00:00:00+00:002014-04-18T00:00:00+00:00
Unknown
https://bloerg.net/posts/re-advanced-vim-tips/<p>Matthias Beyer published <a href="http://beyermatthias.de/advanced-vim-tips/">two tips</a> which according to him “are very critical
when talking about speed when using vim”. I beg to differ and want to give my
reasons.</p>
<p>Yes, leader mappings are important but not to gain speed but to increase the
available mapping space by one additional dimension. For example, I map
<code>:Make<CR></code> to <code>F6</code>, whereas he maps it to <code><Leader>m</code>. Speed gain in my
opinion: zero. Moreover, there is some truth in it that “<a href="https://github.com/AndrewRadev/splitjoin.vim/issues/14">leader maps are pretty
lame</a>”.</p>
<p>Matthias’ next tip was: “don’t use the arrow keys!”. I agree but replacing them
by using <code>hjkl</code> exclusively won’t cut it. You gain a little bit by staying in
the home row but at the end of the day you still drag your cursor around one
character/line after another. By far, my main navigation speedups come from
moving by words with <kbd>b</kbd>, <kbd>e</kbd> and <kbd>w</kbd>, by paragraphs
with <kbd>{</kbd> and <kbd>}</kbd>, by half-screens with
<kbd>Ctrl</kbd>+<kbd>u</kbd> and <kbd>Ctrl</kbd>+<kbd>d</kbd>. However,
navigation only makes up a tiny fraction of a typical Vim day: I gain a lot more
by using Vim’s <a href="http://stevelosh.com/blog/2010/09/coming-home-to-vim/#a-language-of-text-editing">grammar</a> to its full extent.</p>
<p>So here is my Vim tip of the day: <em>learn it as good as you can</em>.</p>
Navigating code with Vim and ctags2014-04-15T00:00:00+00:002014-04-15T00:00:00+00:00
Unknown
https://bloerg.net/posts/navigating-code-with-vim-and-ctags/<p>Vim’s builtin tag integration is an incredibly easy way to jump from one source
location to the definition of a function or class using
<kbd>Ctrl</kbd>+<kbd>]</kbd><sup class="footnote-reference"><a href="#1">1</a></sup> and <kbd>Ctrl</kbd>+<kbd>t</kbd>. To generate
the tags file, I used to map the following command to <code><Leader>gt</code></p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>:!ctags -R -f .tags --sort=yes --exclude=build --exclude=_build
</span></code></pre>
<p>As you probably can imagine, such command line will never cover all corner cases
of files that need to be ex- or included. However, Git already knows about which
files I’d like to ignore, so I now just feed the list of files already versioned
to ctags which I map like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>:!git ls-tree -r --name-only $(git rev-parse --abbrev-ref HEAD) <bar> ctags -f .tags --sort=yes -L -
</span></code></pre>
<p>To open and close the whole folds of target destinations prior and after the
jumps, I use the following functions and maps:</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#668f14;">function </span><span style="color:#c23f31;">TagJumpForward</span><span>()
</span><span> </span><span style="color:#b39f04;">execute </span><span style="color:#d07711;">"tag " </span><span style="color:#668f14;">. </span><span style="color:#b39f04;">expand</span><span>(</span><span style="color:#d07711;">"<cword>"</span><span>)
</span><span> </span><span style="color:#72ab00;">try</span><span> | foldopen! | </span><span style="color:#72ab00;">catch</span><span> | | </span><span style="color:#72ab00;">endtry
</span><span style="color:#668f14;">endfunction
</span><span>
</span><span style="color:#668f14;">function </span><span style="color:#c23f31;">TagJumpBack</span><span>()
</span><span> </span><span style="color:#72ab00;">try</span><span> | foldclose! | </span><span style="color:#72ab00;">catch</span><span> | | </span><span style="color:#72ab00;">endtry
</span><span> pop
</span><span style="color:#668f14;">endfunction
</span><span>
</span><span style="color:#b39f04;">nnoremap </span><span style="color:#a2a001;"><silent> <C-i></span><span> :call </span><span style="color:#b39f04;">TagJumpForward</span><span>()</span><span style="color:#a2a001;"><CR>
</span><span style="color:#b39f04;">nnoremap </span><span style="color:#a2a001;"><silent> <C-t></span><span> :call </span><span style="color:#b39f04;">TagJumpBack</span><span>()</span><span style="color:#a2a001;"><CR>
</span></code></pre>
<p>Finally, if you use CtrlP for buffer and file navigation, you should enable the
tag support and map the launcher, for example to <kbd>Ctrl</kbd>+<kbd>B</kbd></p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">let </span><span style="color:#5597d6;">g:ctrlp_extensions</span><span> = [</span><span style="color:#d07711;">'tag'</span><span>]
</span><span style="color:#b39f04;">nnoremap </span><span style="color:#a2a001;"><C-b></span><span> :CtrlPTag</span><span style="color:#a2a001;"><CR>
</span></code></pre>
<p>That reduces navigations like <code><C-p>foo_file_c<CR>/bar_func n n n</code> to
<code><C-b>bar_func</code>.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p><kbd>Ctrl</kbd>+<kbd>]</kbd> is pretty hard to reach on a German keyboard layout, so I mapped that to <kbd>Ctrl</kbd>+<kbd>i</kbd>.</p>
</div>
On a personal note: keybase.io2014-04-11T00:00:00+00:002014-04-11T00:00:00+00:00
Unknown
https://bloerg.net/posts/on-a-personal-note-keybase-io/<p>A <a href="http://blog.lrdesign.com/2014/03/thoughts-on-keybase-io/">lot</a>
<a href="http://www.tbray.org/ongoing/When/201x/2014/03/19/Keybase">has</a>
<a href="https://news.ycombinator.com/item?id=7502730">been</a> <a href="http://mspr0.de/?p=4078">written</a> about
<a href="https://keybase.io">Keybase.io</a>, a public directory of public encryption keys
associated with usernames and additional “verification proofs”. Yesterday, I was
admitted to the circle of alpha testers and could grab the
<a href="https://keybase.io/matze">keybase.io/matze</a> user name. Despite some initial
<a href="https://github.com/keybase/keybase-issues/issues/556">problems</a>, the website itself works just fine. Using the keybase client
turned out to be a dead end though. It requires a fairly recent NodeJS and npm
which are both not packaged for my long-term choice of Linux distribution. Thus,
right now, the service offered to me is quite limited, because I am also not
willing to upload my client-encrypted key.</p>
<hr />
<p>I have four invites left, so if you are interested in trying that service, just
drop me a line. And yes, I have a Twitter account for five years now, but this
is and (probably will) be the last occasion that I use it.</p>
Visualizing trace profile data2014-04-07T00:00:00+00:002014-04-07T00:00:00+00:00
Unknown
https://bloerg.net/posts/visualizing-trace-profile-data/<p>To analyze the run-time behaviour of an application, a common technique is to record
traces of code execution by inserting statements like these:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#5597d6;">start_trace </span><span>(</span><span style="color:#d07711;">"foo"</span><span>);
</span><span style="color:#5597d6;">do_foo </span><span>();
</span><span style="color:#5597d6;">end_trace </span><span>(</span><span style="color:#d07711;">"foo"</span><span>);
</span></code></pre>
<p>This is an effective way to analyze concurrent applications that are usually
difficult to reason about. If the IDE<sup class="footnote-reference"><a href="#1">1</a></sup> does not support it, visualizing this
kind of data can be tricky. Fortunately, the Chrome browser exposes its internal
trace viewer via a generic [JSON format][] interface. All you have to do is
generate the appropriate data and load it in the <code>about:tracing</code> page:</p>
<figure>
<img src="/images/2014-04-07/vis.png" />
</figure>
<p>Here is a very simple way how you could trace the execution of Python code
without interspersing your code like stupid:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">import </span><span>os
</span><span style="color:#72ab00;">import </span><span>time
</span><span style="color:#72ab00;">import </span><span>threading
</span><span style="color:#72ab00;">import </span><span>functools
</span><span style="color:#72ab00;">import </span><span>json
</span><span>
</span><span>
</span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">Manager</span><span>(</span><span style="font-style:italic;color:#b06936;">object</span><span>):
</span><span>
</span><span> </span><span style="color:#5597d6;">_START_CLOCK </span><span style="color:#72ab00;">= </span><span>time.</span><span style="color:#5597d6;">time</span><span>()
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#b39f04;">__init__</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> </span><span style="color:#5597d6;">self</span><span>._events </span><span style="color:#72ab00;">= </span><span>[]
</span><span> </span><span style="color:#5597d6;">self</span><span>._pid </span><span style="color:#72ab00;">= </span><span>os.</span><span style="color:#5597d6;">getpid</span><span>()
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">_new_event</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">func</span><span>, </span><span style="color:#5597d6;">event</span><span>):
</span><span> tid </span><span style="color:#72ab00;">= </span><span>threading.</span><span style="color:#5597d6;">current_thread</span><span>().ident
</span><span> timestamp </span><span style="color:#72ab00;">= </span><span>(time.</span><span style="color:#5597d6;">time</span><span>() </span><span style="color:#72ab00;">- </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">_START_CLOCK</span><span>) </span><span style="color:#72ab00;">* </span><span style="color:#b3933a;">1000 </span><span style="color:#72ab00;">* </span><span style="color:#b3933a;">1000
</span><span> </span><span style="color:#5597d6;">self</span><span>._events.</span><span style="color:#5597d6;">append</span><span>(</span><span style="color:#a2a001;">dict</span><span>(</span><span style="color:#5597d6;">name</span><span style="color:#72ab00;">=</span><span>func.</span><span style="color:#a2a001;">__name__</span><span>,
</span><span> </span><span style="color:#5597d6;">cat</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'f'</span><span>, </span><span style="color:#5597d6;">ph</span><span style="color:#72ab00;">=</span><span>event,
</span><span> </span><span style="color:#5597d6;">ts</span><span style="color:#72ab00;">=</span><span>timestamp,
</span><span> </span><span style="color:#5597d6;">tid</span><span style="color:#72ab00;">=</span><span>tid, </span><span style="color:#5597d6;">pid</span><span style="color:#72ab00;">=</span><span style="color:#5597d6;">self</span><span>._pid))
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">trace</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">func</span><span>):
</span><span> @functools.</span><span style="color:#5597d6;">wraps</span><span>(func)
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">record</span><span>(</span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">args</span><span>, </span><span style="color:#72ab00;">**</span><span style="color:#5597d6;">kwargs</span><span>):
</span><span> </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">_new_event</span><span>(func, </span><span style="color:#d07711;">'B'</span><span>)
</span><span> </span><span style="color:#5597d6;">func</span><span>(</span><span style="color:#72ab00;">*</span><span>args, </span><span style="color:#72ab00;">**</span><span>kwargs)
</span><span> </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">_new_event</span><span>(func, </span><span style="color:#d07711;">'E'</span><span>)
</span><span>
</span><span> </span><span style="color:#72ab00;">return </span><span>record
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#b39f04;">__del__</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> </span><span style="color:#72ab00;">with </span><span style="color:#b39f04;">open</span><span>(</span><span style="color:#d07711;">'trace.json'</span><span>, </span><span style="color:#d07711;">'w'</span><span>) </span><span style="color:#72ab00;">as </span><span>fp:
</span><span> json.</span><span style="color:#5597d6;">dump</span><span>(</span><span style="color:#a2a001;">dict</span><span>(</span><span style="color:#5597d6;">traceEvents</span><span style="color:#72ab00;">=</span><span style="color:#5597d6;">self</span><span>._events), fp)
</span></code></pre>
<p>The <code>Manager</code> records all events for functions that are decorated with the
<code>trace</code> decorator, nothing fancy here. The distinction between the thread id
<code>tid</code> and the process id <code>pid</code> stems from the fact that Chrome has a
multi-process architecture, but you can of course use these fields in any way
you like. Just remember that events from the same thread id are layed out in the
same row and therefore need correctly ordered time stamps. Note that I also
didn’t took special care for the category field, i.e. just set it to “f”. In
the <code>Manager</code>s destructor<sup class="footnote-reference"><a href="#2">2</a></sup> I just dump the events in the correct format.</p>
<p>The following test program demonstrates how to use the <code>Manager</code> and was used to
made the top image:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span>m </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">Manager</span><span>()
</span><span>
</span><span>@m.</span><span style="color:#5597d6;">trace
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">foo</span><span>(</span><span style="color:#5597d6;">t</span><span>):
</span><span> </span><span style="color:#72ab00;">print </span><span style="color:#d07711;">'going to sleep for </span><span style="color:#aeb52b;">{}</span><span style="color:#d07711;"> seconds'</span><span>.</span><span style="color:#5597d6;">format</span><span>(t)
</span><span> time.</span><span style="color:#5597d6;">sleep</span><span>(t)
</span><span>
</span><span>threads </span><span style="color:#72ab00;">= </span><span>[]
</span><span>
</span><span style="color:#72ab00;">for </span><span>i </span><span style="color:#72ab00;">in </span><span style="color:#b39f04;">range</span><span>(</span><span style="color:#b3933a;">25</span><span>):
</span><span> thread </span><span style="color:#72ab00;">= </span><span>threading.</span><span style="color:#5597d6;">Thread</span><span>(</span><span style="color:#5597d6;">target</span><span style="color:#72ab00;">=</span><span>foo, </span><span style="color:#5597d6;">args</span><span style="color:#72ab00;">=</span><span>(random.</span><span style="color:#5597d6;">random</span><span>() </span><span style="color:#72ab00;">* </span><span style="color:#b3933a;">0.25</span><span>,))
</span><span> thread.</span><span style="color:#5597d6;">start</span><span>()
</span><span> threads.</span><span style="color:#5597d6;">append</span><span>(thread)
</span><span> time.</span><span style="color:#5597d6;">sleep</span><span>(random.</span><span style="color:#5597d6;">random</span><span>() </span><span style="color:#72ab00;">* </span><span style="color:#b3933a;">0.025</span><span>)
</span><span>
</span><span style="color:#72ab00;">for </span><span>thread </span><span style="color:#72ab00;">in </span><span>threads:
</span><span> thread.</span><span style="color:#5597d6;">join</span><span>()
</span></code></pre>
<p>So, now stop wasting your time guessing the run-time behaviour of your
application and <em>measure</em> it!</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>As far as I know, Eclipse has a mode to show thread execution.
<sup class="footnote-reference"><a href="#2">2</a></sup>: This pattern is typically frowned upon by seasoned Python hackers but once in a while it can be of a good use.
[JSON format]: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit</p>
</div>
The smallest literate Python implementation2014-03-10T00:00:00+00:002014-03-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/the-smallest-literate-python-implementation/<p><a href="http://en.wikipedia.org/wiki/Literate_programming">Literate programming</a> is a programming paradigm that intermingles natural
language comments with short snippets of source code from which then both
compiled code and documentation can be generated. Knuth’s TeX is probably the
most famous project written in a literate programming style.</p>
<p>Although, I am not a big fan of literate programming myself, I appreciate the
general idea of interweaving natural language with code snippets. For example, I
am always intrigued by explorative documents for research, in which you develop
an idea in words and code. <a href="http://ipython.org/notebook">IPython notebooks</a> come pretty close, but they are
not what I would call plain text.</p>
<p>I was then looking for alternatives that mix and match Markdown with Python code
and output static Markdown or HTML. I found Stijn Debrouwere’s
<a href="https://github.com/stdbrouw/python-literate">python-literate</a>, but it is too focused too much on the literate
<em>programming</em> part and tries to invent basically anything from scratch.</p>
<p>So, I sat down and wrote a tiny script called <a href="https://github.com/matze/pyl">pyl</a> that does exactly one job,
execute Python code and output results. It uses Pandoc to parse Markdown<sup class="footnote-reference"><a href="#1">1</a></sup>
into Pandoc’s JSON <a href="http://en.wikipedia.org/wiki/Abstract_syntax_tree">syntax tree</a> representation, traverses the tree, executes
all code blocks within Python and replaces inline code with defined variables.
It then outputs the altered syntax tree from which a second Pandoc instance can
generate the final output in any desired target format.</p>
<p>With that 30-odd line long Python script, you can turn</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Here is some Python code
</span><span>
</span><span> x = 2 * 3 + 4
</span><span> z = 'A fine string'
</span><span>
</span><span>Now, is `x` = 10? and `z` a fine string?
</span></code></pre>
<p>into this</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Here is some Python code
</span><span>
</span><span> x = 2 * 3 + 4
</span><span> z = 'A fine string'
</span><span>
</span><span>Now, is 10 = 10? and A fine string a fine string?
</span></code></pre>
<p>Because there are no restrictions, you can do all kinds of crazy and potentially
dangerous things. Just have a look at the <a href="https://github.com/matze/pyl/tree/master/examples">example</a> gallery.</p>
<p><em>Again, one word of warning</em>: pyl executes arbitrary code in a non-sandboxed Python
environment. This gives a lot of opportunities but can be a major security
problem if building documents from untrusted sources.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Actually anything that Pandoc can parse should work with pyl, but I haven’t checked that yet.</p>
</div>
Extending Git2014-02-20T00:00:00+00:002014-02-20T00:00:00+00:00
Unknown
https://bloerg.net/posts/extending-git/<p>As a follow-up to my cherry pick UI <a href="/2013/04/25/a-git-cherry-pick-text-user-interface.html">post</a>, here’s a quick tutorial how you
can extend Git with new sub-commands. If you used the cherry pick tool, you
noticed that it is not very well integrated and you needed to remember the exact
name of the script. I suggested to add an alias to your <code>.gitconfig</code> but there
is more “natural” way to do it.</p>
<p>Luckily<sup class="footnote-reference"><a href="#1">1</a></sup>, Git uses any program prefixed with <code>git-</code> and located in <code>$PATH</code> as
a sub-command. That means, moving the <code>pick-from</code> to <code>/usr/local/bin/git-pick</code>,
allows us to use it with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ git pick <feature-branch>
</span></code></pre>
<p>Calling <code>git pick --help</code> spawns <code>man</code> for a <code>git-pick.7</code> manpage. Hence, you
could for example generate one from Markdown with Pandoc and copy that into your
local man search directory <code>~/.local/share/man/man7</code>.</p>
<p>Git’s sub-command completion is implemented in a similar pragmatic way: simply
provide a bash function <code>_git_command_name()</code> that completes to whatever you
like. Git already provides helper functions for typical situations, which is why
we can complete branches for <code>git pick</code> simply with</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#668f14;">function </span><span style="color:#c23f31;">_git_pick </span><span>() {
</span><span> </span><span style="color:#5597d6;">__gitcomp_nl </span><span style="color:#d07711;">"$(</span><span style="color:#acb3c2;">__git_refs</span><span style="color:#d07711;">)"
</span><span>}
</span></code></pre>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>… and probably a reminiscence of Git’s past as a huge collection of porcelain Perl scripts around the Git core.</p>
</div>
Virtual environments for compiled libraries and applications2014-02-12T00:00:00+00:002014-02-12T00:00:00+00:00
Unknown
https://bloerg.net/posts/virtual-environments-for-compiled-libraries-and-applications/<p>Compiling software in a restricted environment requires changing essential
environment variables and finding a place where to install the final product.
This is a tedious task that can be mitigated by a simple <a href="https://github.com/matze/configs/blob/master/.local/bin/cenv">shell script</a> that
sets up the appropriate directory and overrides the path variables:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ cenv foo
</span><span>$ (foo) ./configure --prefix=$CENV && make && make install
</span></code></pre>
<p>Because this script only knows about a limited set of environment variables, you
can add more by running</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ cenv --edit foo
</span></code></pre>
<p>This will open a local file that is sourced right before starting the new
subshell and in which you can define more variables to be exported.</p>
<p>To make it work even more like a Python virtualenv, you could alias common build
tools in your <code>.bashrc</code> to include the prefix implicitly:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#b39f04;">alias </span><span style="color:#c23f31;">cmake</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"cmake -DCMAKE_INSTALL_PREFIX=$</span><span style="color:#acb3c2;">CENV</span><span style="color:#d07711;">"
</span><span style="color:#b39f04;">alias </span><span style="color:#c23f31;">configure</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"configure --prefix=$</span><span style="color:#acb3c2;">CENV</span><span style="color:#d07711;">"
</span></code></pre>
<p>This little script works for my use case but there are of course alternatives:
lightweight containers (either raw or using Docker) or virtual machines being
some of them.</p>
A sane beamer directory structure2014-01-23T00:00:00+00:002014-01-23T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-sane-beamer-directory-structure/<p>I had to give many talks, both to internal and external audiences. As you could
imagine, I use LaTeX and Beamer to compile my slides. But until now, I had no
good approach how to organize them. This lead to a chaos of copy-pasted
Makefiles, assets such as images and TikZ sources, TeX preludes etc. Everything
was scattered around several Git repositories, some talks have their own
repositories, some were collected in a single one. That was quite a mess, to be
honest.</p>
<p>So, I sat down and thought how to lay out things. Eventually, I set up one
repository for all Beamer sources which are simply structured in a year/title
hierarchy. I set up one repository for all assets and one repository for my
BibTeX databases. At the moment I don’t have a good reason to split them that
way other than that it doesn’t feel “semantically” right. To avoid repeating
myself in the prelude, I pulled out the most common settings – including
packages, fonts, colors, environments and commands – into its own top-level
<code>beamercommon.sty</code>.</p>
<p>Each talk subdirectory as its own Makefile that consists of one of:</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#72ab00;">include </span><span style="color:#d07711;">../../common.mk
</span><span style="color:#72ab00;">include </span><span style="color:#d07711;">../../common.pandoc.mk
</span></code></pre>
<p>The first Makefile is based on <a href="/2013/06/25/use-rubber-to-compile-tex.html">rubber</a> and simply tries to compile all <code>.tex</code>
files found in the current directory. The second Makefile uses <a href="/2012/12/05/pandoc-and-tex.html">Pandoc</a> and a
slightly modified template to build Beamer slides from Markdown input:</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#5597d6;">SRC</span><span style="color:#72ab00;">=$(</span><span style="color:#b39f04;">wildcard </span><span style="color:#acb3c2;">*</span><span style="color:#d07711;">.md</span><span style="color:#72ab00;">)
</span><span style="color:#5597d6;">PDF</span><span style="color:#72ab00;">=$(</span><span style="color:#b39f04;">subst </span><span style="color:#d07711;">.md,.pdf,</span><span style="color:#72ab00;">$(</span><span style="color:#acb3c2;">SRC</span><span style="color:#72ab00;">))
</span><span style="color:#5597d6;">TEMPLATE</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">../../pandoc-template.tex
</span><span style="color:#5597d6;">THEME_OPTIONS</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"english, titlepage0"
</span><span style="color:#5597d6;">OPTS</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">-t beamer --latex-engine=xelatex --template=</span><span style="color:#72ab00;">$(</span><span style="color:#acb3c2;">TEMPLATE</span><span style="color:#72ab00;">) </span><span style="color:#d07711;">\
</span><span style="color:#d07711;"> --variable theme=KIT \
</span><span style="color:#d07711;"> --variable sansfont="Source Sans Pro" \
</span><span style="color:#d07711;"> --variable monofont="Inconsolata" \
</span><span style="color:#d07711;"> --variable fontsize=18pt \
</span><span style="color:#d07711;"> --variable graphicspath=../../common/images/ \
</span><span style="color:#d07711;"> --variable themeoptions=</span><span style="color:#72ab00;">$(</span><span style="color:#acb3c2;">THEME_OPTIONS</span><span style="color:#72ab00;">) </span><span style="color:#d07711;">\
</span><span>
</span><span style="color:#c23f31;">.PHONY</span><span style="color:#72ab00;">: </span><span style="color:#d07711;">clean
</span><span>
</span><span style="color:#c23f31;">all</span><span style="color:#72ab00;">: $(</span><span style="color:#acb3c2;">PDF</span><span style="color:#72ab00;">)
</span><span>
</span><span style="color:#5597d6;">%</span><span style="color:#c23f31;">.pdf</span><span style="color:#72ab00;">: </span><span style="color:#acb3c2;">%</span><span style="color:#d07711;">.md </span><span style="color:#72ab00;">$(</span><span style="color:#acb3c2;">TEMPLATE</span><span style="color:#72ab00;">)
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">pandoc </span><span style="color:#72ab00;">$(</span><span style="color:#5597d6;">OPTS</span><span style="color:#72ab00;">)</span><span style="color:#5597d6;"> -o $@ $<
</span><span>
</span><span style="color:#c23f31;">clean</span><span style="color:#72ab00;">:
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">rm -f </span><span style="color:#72ab00;">$(</span><span style="color:#5597d6;">PDF</span><span style="color:#72ab00;">)
</span></code></pre>
<p>In the talk-specific Makefile, I can then simply add or override the <code>$(OPTS)</code>
variable to customize the outcome.</p>
<p>There was one thing, I wasn’t sure about: keeping the common and BibTeX repos
outside, inside as a submodule or inside as a subtree. I decided for the
submodule approach, although I know, it could hurt me in the long run. In any
case, this is what the final, version-controlled directory structure looks like:</p>
<figure>
<img src="/images/2014-23-01/structure.png" />
</figure>
Easy Jekyll post drafting2014-01-08T00:00:00+00:002014-01-08T00:00:00+00:00
Unknown
https://bloerg.net/posts/easy-jekyll-post-drafting/<p><a href="http://jekyllrb.com">Jekyll</a>, the static site compiler powering this blog, comes with a nice draft
feature for some time now. Draft posts are placed into the <code>_drafts</code> directory
instead of the normal <code>_posts</code> directory and built and served only if a special
flag is passed to Jekyll. After using this for some time now, it became a real
chore to name and rename the files according to the proper Jekyll conventions
(e.g. <code>_posts/2013-23-12-foo-bar-baz.md</code>). Of course, I had to do something
about it …</p>
<p>Using <a href="https://github.com/matze/configs/blob/master/.bin/jk">this</a> tiny script, I can now create regular posts or drafts with the
YAML header simply by writing</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ jk post A post about this
</span><span>$ jk draft A draft about that
</span></code></pre>
<p>Furthermore, I can move posts between both directories, identifying the file in
question via a simple fuzzy search<sup class="footnote-reference"><a href="#1">1</a></sup>, e.g.</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ jk switch about that
</span><span>Moved _drafts/2013-12-30-a-draft-about-that.md -> _posts/2014-01-08-a-draft-about-that.md
</span></code></pre>
<p>moves the post to the drafts directory. The same mechanics is used to edit a
file, e.g. Vim is launched for the draft if, I run</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ jk edit about that
</span></code></pre>
<p>That’s it.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>The implementation is questionable but it works fine for me.</p>
</div>
A tiny Vim plugin for interactive Git rebasing2013-12-04T00:00:00+00:002013-12-04T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-tiny-vim-plugin-for-interactive-git-rebasing/<p>One of Git’s killer features is <a href="http://git-scm.com/book/en/Git-Tools-Rewriting-History#Changing-Multiple-Commit-Messages">interactive rebase</a>. After choosing a
range of commits and rebasing them interactively with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>git rebase -i HEAD~10
</span></code></pre>
<p>you can re-order, edit, drop and squash commits, simply by fiddling with the
list of commits pasted into your favorite editor. Some actions are easily done
in Vim (e.g. deleting or moving lines), others require writing whatever you
intend to do with the commit. Because I squash a lot, I became tired of typing
<code>^cws</code> over and over again and finally sat down to write this stupid little
filetype plugin:</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#668f14;">function </span><span style="color:#c23f31;">RebaseActionToggle</span><span>()
</span><span> </span><span style="color:#b39f04;">let</span><span> line = </span><span style="color:#b39f04;">getline</span><span>(</span><span style="color:#d07711;">"."</span><span>)
</span><span> </span><span style="color:#b39f04;">let</span><span> result = </span><span style="color:#b39f04;">matchstr</span><span>(line, </span><span style="color:#d07711;">"^\\a"</span><span>)
</span><span> </span><span style="color:#b39f04;">let</span><span> transitions = {</span><span style="color:#d07711;">'p'</span><span>: </span><span style="color:#d07711;">'squash'</span><span>, </span><span style="color:#d07711;">'s'</span><span>: </span><span style="color:#d07711;">'edit'</span><span>, </span><span style="color:#d07711;">'e'</span><span>: </span><span style="color:#d07711;">'fixup'</span><span>, </span><span style="color:#d07711;">'f'</span><span>: </span><span style="color:#d07711;">'pick'</span><span>}
</span><span> </span><span style="color:#b39f04;">execute </span><span style="color:#d07711;">"normal! ^cw" </span><span style="color:#668f14;">.</span><span> transitions[result]
</span><span style="color:#668f14;">endfunction
</span><span>
</span><span style="color:#b39f04;">noremap </span><span style="color:#a2a001;"><Cr></span><span> :call </span><span style="color:#b39f04;">RebaseActionToggle</span><span>()</span><span style="color:#a2a001;"><Cr>
</span></code></pre>
<p>If you save it as <code>gitrebase.vim</code> in <code>.vim/after/ftplugin</code>, you can cycle
through the different rebase actions by pressing <kbd>Enter</kbd>:</p>
<figure><img src="http://i.imgur.com/13A7WbL.gif"></figure>
<p>Have fun with that.</p>
You know why GitHub fucking rocks?2013-11-15T00:00:00+00:002013-11-15T00:00:00+00:00
Unknown
https://bloerg.net/posts/you-know-why-github-fucking-rocks/<p>No, not only because it’s “social coding” done right. But because the ecosphere
around GitHub is alive and thriving. We have <a href="https://travis-ci.org">Travis CI</a> to build around the
clock. We have <a href="https://readthedocs.org">Read the Docs</a> to re-generate Sphinx documentation on each
push. We have <a href="https://coveralls.io">Coveralls.io</a> to check the test coverage and we will have
<a href="http://gitter.im">gitter</a> to have instant, per-repo support chat.</p>
<p>Yes everything is in the “cloud” and yes I could setup local alternatives. But I
don’t want to care about infrastructure. I want to churn out commit after
commit. Thanks everyone for making that a slick experience.</p>
Turning CMake into a good UNIX citizen2013-10-29T00:00:00+00:002013-10-29T00:00:00+00:00
Unknown
https://bloerg.net/posts/turning-cmake-into-a-good-unix-citizen/<p>CMake, the software build system, describes itself as:</p>
<blockquote>
<p>Welcome to CMake, the cross-platform, open-source build system. CMake is a
family of tools designed to build, test and package software.</p>
</blockquote>
<p>However, CMake cannot keep its promises and fails with respect to packaging<sup class="footnote-reference"><a href="#1">1</a></sup>.
Writing software with CMake that builds everywhere is possible but in some cases
not very easy and, although CMake provides its own packaging mechanism called
CPack, it can only serve as a temporary solution.</p>
<p>The main problem on UNIX-like systems is the amount of freedom concerning
installation paths. There are no guidelines where things have to go and how the
paths are set up. Yes, CMake allows you to set the installation prefix by
using</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>cmake . -DCMAKE_INSTALL_PREFIX=/usr
</span></code></pre>
<p>however, this is not enough to cover all cases. For example, the location of a
64-bit library is <code>/usr/lib</code> on Debian systems and <code>/usr/lib64</code> on openSUSE
systems. In an autotools-based system, you would just set <code>--libdir</code> to the
appropriate location. Not so with CMake. People write all kinds of gruesome
hacks to figure this out at build time much to mental state of package
maintainers. Moreover, even if one user gets this whole mess right, a thousand
others won’t.</p>
<p>To fix this problem for my own software systems, I wrote the
<a href="https://gist.github.com/matze/7200903">ConfigurePaths</a> module which would be
used like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>include(ConfigurePaths)
</span><span>
</span><span>configure_paths(FOO)
</span></code></pre>
<p>Simple, eh? With this call, you populated your global CMake namespace with
cached path variables that mimic the system variables known from <code>configure</code>
scripts:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span># header files
</span><span>install(FILES bar.h
</span><span> DESTINATION ${FOO_INCLUDEDIR})
</span><span>
</span><span># libraries
</span><span>install(TARGETS bar
</span><span> LIBRARY DESTINATION ${FOO_LIBDIR})
</span><span>
</span><span># binaries
</span><span>install(TARGETS baruser
</span><span> RUNTIME DESTINATION ${FOO_BINDIR})
</span><span>
</span><span># pkg-config
</span><span>install(FILES bar.pc
</span><span> DESTINATION ${FOO_PKGCONFIGDIR})
</span><span>
</span><span># ...
</span></code></pre>
<p>By default, all variables are set to the <code>configure</code> defaults, e.g. <code>FOO_PREFIX</code>
is <code>/usr/local</code>, <code>FOO_DATADIR</code> is <code>FOO_PREFIX/share</code>, and so on. The user can
override the default values by CMake means, for example:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ cmake . -DPREFIX=/opt -DLIBDIR=/usr/lib64
</span></code></pre>
<p>Rather than convincing you to use this, I really just want to encourage you to
think about the whole mess. In the end, it may turn out that <em>you</em> have to
package the software yourself.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>I deliberately do not bash the ugly syntax here. Or say that autotools is a much better build tool on UNIX systems. No, no, I won’t.</p>
</div>
Git and OpenCL introductory slides2013-10-18T00:00:00+00:002013-10-18T00:00:00+00:00
Unknown
https://bloerg.net/posts/introductory-git-and-opencl-slides/<p>Today was the last day of a workshop organized by the <a href="http://www.kseta.kit.edu/index.php">KSETA</a> grad school.
It’s an interdisclipinary grad school focused mainly towards theoretical and
experimental physicists but also enrolls engineering and computing students.
All participants had to prepare an introductional tutorial-like talk either
about some technology or something related to their day-to-day work.</p>
<p>I decided for the first option and gave two talks: one on <a href="https://speakerdeck.com/matze/distributed-version-control-and-why-you-want-to-use-it">Git</a>, the other on
<a href="https://speakerdeck.com/matze/gpgpu-computing-with-opencl">OpenCL</a>. My main goal was to tell everyone, how these tools can make life
easier and that these technologies are nothing to be scared off. Both talks were
aimed at total beginners and – still to my own surprise – well received.</p>
<p>If you want to use, extend or otherwise redistribute the content, you are free
to do so. Both slide sets are hereby published under CC-BY-SA-3.0 with their
sources available <a href="https://github.com/matze/kseta-dvcs-talk">here</a> and
<a href="https://github.com/matze/kseta-opencl-talk">here</a>. Note, that you probably have
to adjust the theme and maybe also the font, if you do not like Adobe’s
<em>Source Sans Pro</em>.</p>
LaTeX package installation with tex-get2013-10-14T00:00:00+00:002013-10-14T00:00:00+00:00
Unknown
https://bloerg.net/posts/latex-package-installation-with-tex-get/<p>If you are running a Linux system, you probably install your TeX Live
distribution from you Linux distribution’s package manager. The current Ubuntu
12.04 LTS ships TeX Live 2009 which is … old and a treasure trove of TeX
stackexchange questions.</p>
<p>A common answer to questions like “how can I install package XYZ locally?” is to
install them into the local <code>texmf</code> tree“. The TeX Live distribution comes with
its own package manager called <code>tlmgr</code> but Debian decided to drop it in favor of
their own packaging system. So, one other option to update or install missing
packages, is populating a local <code>texmf</code> tree.</p>
<p>The installation is actually quite simple except that packages litter their
files all over the place; a problem that was recently solved in the Vim
community by run-time managers such as Pathogen or Vundle. So, how would you do
it? First, grab the zipped package from CTAN and extract it somewhere. Now you
need to find out where your <code>texmf</code> tree is located with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>kpsewhich -var-value=TEXMFHOME
</span></code></pre>
<p>and put the .sty, .bst etc. files in the respective subdirectories. The LaTeX
<a href="http://en.wikibooks.org/wiki/LaTeX/Installing_Extra_Packages#Manual_installation">Wikibook</a> gives a good overview, what belongs where. Although you could place
the files in the top-level directories (e.g. all .sty files in
<code>~/texmf/tex/latex</code>) it’s a better idea to separate things into package
subdirectories. If there are no .sty files, you might need to generate them by
running LaTeX on the .ind files and install the generated files. You now have
fully working manually maintained packages but managing more than one package
will become a real chore eventually.</p>
<p>To help me with all the tedious tasks, I wrote a little shell script called
<a href="https://github.com/matze/configs/blob/master/.bin/tex-get">tex-get</a>. It checks if any files need to be generated and places
everything into their respective destination directories. While doing so, all
installed filenames are recorded in a manifest file, which is then used for
removal of the package. The commandline interface is similar to <code>apt-get</code>, hence
you can install and remove packages with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>tex-get install l3kernel
</span><span>tex-get remove l3kernel
</span></code></pre>
<p>However, unlike a <em>real</em> package system, dependencies are not resolved.
Moreover, no documentation is generated and no font files are (yet) installed.
Anyway, suggestions and improvements are very welcome.</p>
Syncing mails with mbsync (instead of OfflineIMAP)2013-10-09T00:00:00+00:002013-10-09T00:00:00+00:00
Unknown
https://bloerg.net/posts/syncing-mails-with-mbsync-instead-of-offlineimap/<p><em>Update</em>: Ryan Lue pointed me to his
<a href="https://github.com/rlue/little_red_flag">little_red_flag</a> daemon which triggers
mbsync synchronization if changes happen both locally <em>and</em> remotely. Try it
out!</p>
<p>A lot of people recommend OfflineIMAP for syncing mails between a remote IMAP
account and a local maildir mailbox for mutt<sup class="footnote-reference"><a href="#1">1</a></sup> pleasure. However,
OfflineIMAP’s quality is far from what I think is necessary for a tool handling
sensitive data such as mail. Looking for an alternative, I stumbled upon
[mbsync][] which was maintained by Ted Ts’o for a while. Setting it up to
synchronize with the Google mail servers is pretty simple once you know the
quirks of Google’s IMAP implementation and mbsync’s weird configuration
approach.</p>
<p>To get it working with Google I had to compile mbsync from source — the
packaged version didn’t work for me. mbsync just wouldn’t download any mails.
So, go ahead, download the mbsync source and <code>configure && make && make install</code>
it. mbsync groups its <code>.mbsyncrc</code> configuration into account details, stores and
channels that connect stores for synchronization. A typical GMail account would
look like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>IMAPAccount gmail
</span><span>Host imap.gmail.com
</span><span>User foo.bar@gmail.com
</span><span>Pass secret
</span><span>UseIMAPS yes
</span><span>CertificateFile /etc/ssl/certs/ca-certificates.crt
</span></code></pre>
<p>Saving the password plaintext is not the best idea. Fortunately, the latest
mbsync sources come with the <code>PassCmd</code> setting, which is just a command that is
run in a subprocess of mbsync. With a tiny Python script such as this:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#7f8989;">#!/usr/bin/env python
</span><span>
</span><span style="color:#72ab00;">import </span><span>argparse
</span><span style="color:#72ab00;">import </span><span>keyring
</span><span style="color:#72ab00;">import </span><span>getpass
</span><span>
</span><span style="color:#72ab00;">if </span><span style="color:#a2a001;">__name__ </span><span style="color:#72ab00;">== </span><span style="color:#d07711;">'__main__'</span><span>:
</span><span> </span><span style="color:#5597d6;">SERVICE </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">'mbsync'
</span><span>
</span><span> parser </span><span style="color:#72ab00;">= </span><span>argparse.</span><span style="color:#5597d6;">ArgumentParser</span><span>()
</span><span> group </span><span style="color:#72ab00;">= </span><span>parser.</span><span style="color:#5597d6;">add_mutually_exclusive_group</span><span>(</span><span style="color:#5597d6;">required</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">True</span><span>)
</span><span> group.</span><span style="color:#5597d6;">add_argument</span><span>(</span><span style="color:#d07711;">'--set'</span><span>, </span><span style="color:#d07711;">'-s'</span><span>, </span><span style="color:#5597d6;">type</span><span style="color:#72ab00;">=</span><span style="color:#a2a001;">str</span><span>, </span><span style="color:#5597d6;">help</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'Account to save password'</span><span>)
</span><span> group.</span><span style="color:#5597d6;">add_argument</span><span>(</span><span style="color:#d07711;">'--get'</span><span>, </span><span style="color:#d07711;">'-g'</span><span>, </span><span style="color:#5597d6;">type</span><span style="color:#72ab00;">=</span><span style="color:#a2a001;">str</span><span>, </span><span style="color:#5597d6;">help</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'Account to get password'</span><span>)
</span><span>
</span><span> args </span><span style="color:#72ab00;">= </span><span>parser.</span><span style="color:#5597d6;">parse_args</span><span>()
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>args.set:
</span><span> password </span><span style="color:#72ab00;">= </span><span>getpass.</span><span style="color:#5597d6;">getpass</span><span>()
</span><span> keyring.</span><span style="color:#5597d6;">set_password</span><span>(</span><span style="color:#5597d6;">SERVICE</span><span>, args.set, password)
</span><span> </span><span style="color:#72ab00;">else</span><span>:
</span><span> </span><span style="color:#b39f04;">print</span><span>(keyring.</span><span style="color:#5597d6;">get_password</span><span>(</span><span style="color:#5597d6;">SERVICE</span><span>, args.get))
</span></code></pre>
<p>and <code>PassCmd</code> set to:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>PassCmd "imap-pass -g foo.bar@gmail.com"
</span></code></pre>
<p>we have a relatively secure way of retrieving the login password through our
system’s keyring. Now we have to define a proxy store for the IMAP account and a
store for our local maildir</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>IMAPStore gmail-remote
</span><span>Account gmail
</span><span>
</span><span>MaildirStore gmail-local
</span><span>Path ~/mail/gmail/
</span><span>Inbox ~/mail/gmail/INBOX
</span></code></pre>
<p>To sync between those stores, you have to set up a channel. Most options are
self-explanatory, however you must take care to use the correct label names for
the sent, draft, starred etc. mail. The names depend on the language set in the
web interface! For me, the verbose flag <code>-V</code> came in handy because this displays
the communication with the IMAP server, including listing all the available IMAP
directories.</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Channel gmail
</span><span>Master :gmail-remote:
</span><span>Slave :gmail-local:
</span><span>Patterns "INBOX" "[Gmail]/Sent Mail" "[Gmail]/Drafts" "[Gmail]/Starred"
</span><span>Create Both
</span><span>Expunge Both
</span><span>SyncState *
</span></code></pre>
<p>Now, running <code>mbsync gmail</code> syncs mail on the Google servers with your local
maildir in <code>~/mail/gmail</code>.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>I wanted to write a post about mutt for quite a while now, but I am not ready with my configuration nor do I have much time to document my progress. Hence, I will present different aspects one by one.
[mbsync]: http://isync.sourceforge.net/mbsync.html</p>
</div>
Updates in Vim Land 20132013-09-22T00:00:00+00:002013-09-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/updates-in-vim-land-2013/<p>More than a year has passed since I <a href="/2012/08/23/updates-in-vim-land.html">wrote</a> about updates of my Vim
configuration. During that year, several changes accumulated concerning package
management, folds, mappings and general organization of my <code>.vimrc</code>.</p>
<h2 id="vundle-becomes-neobundle">Vundle becomes NeoBundle</h2>
<p>Vundle worked very well over the last year, but sometimes you want to pin a
particular revision or branch of a repository. For instance, I keep a fork of
the <a href="https://github.com/matze/vim-markdown-folding">vim-markdown-folding</a> plugin to prevent the plugin from setting the
foldtext variable. With Shougo’s <a href="https://github.com/Shougo/neobundle.vim">NeoBundle.vim</a> plugin, I can simply specify
my forked branch without messing with master. To switch from Vundle to
NeoBundle add</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#72ab00;">if </span><span style="color:#b39f04;">has</span><span>(</span><span style="color:#d07711;">'vim_starting'</span><span>)
</span><span> </span><span style="color:#b39f04;">set</span><span> rtp+</span><span style="color:#668f14;">=~</span><span style="color:#c49a39;">/.vim/</span><span>bundle/neobundle</span><span style="color:#668f14;">.</span><span>vim
</span><span style="color:#72ab00;">endif
</span><span>
</span><span>call </span><span style="color:#b39f04;">neobundle#rc</span><span>(</span><span style="color:#b39f04;">expand</span><span>(</span><span style="color:#d07711;">'~/.vim/bundle/'</span><span>))
</span></code></pre>
<p>on top and replace Bundle with NeoBundle</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#668f14;">...
</span><span>NeoBundle </span><span style="color:#d07711;">'matze/vim-markdown-folding'</span><span>, </span><span style="color:#d07711;">'user-defined-foldtext'
</span><span>NeoBundle </span><span style="color:#d07711;">'mileszs/ack.vim'
</span><span>NeoBundle </span><span style="color:#d07711;">'nanotech/jellybeans.vim'
</span><span style="color:#668f14;">...
</span></code></pre>
<p>There is one shortcoming compared to Vundle though: you won’t see a nice status
and summary page when running <code>:NeoBundleUpdate</code> ☹.</p>
<h2 id="powerline-becomes-airline">Powerline becomes Airline</h2>
<p>The author of the original <a href="https://github.com/Lokaltog/vim-powerline">vim-powerline</a> plugin deprecated it in favor of
Python-based <a href="https://github.com/Lokaltog/powerline">powerline</a> that covers much more than just Vim. Unfortunately,
the old plugin isn’t maintained any more<sup class="footnote-reference"><a href="#1">1</a></sup> and the new one is a big project
with currently 192 open issues and 25 open pull requests.</p>
<p>Fortunately, Bailey Ling stepped up and wrote a complete replacement called
<a href="https://github.com/bling/vim-airline">vim-airline</a> that is lighter than powerline and written in pure Vimscript.
I’ll spare you a screenshot as the look itself didn’t really change.</p>
<h2 id="ag-is-better-than-ack-is-better-than-grep">ag is better than ack is better than grep</h2>
<p>Years ago I switched from regular grep to <a href="http://beyondgrep.com/">ack</a> for typical searches because
it is faster, has saner defaults and doesn’t search files and directories I am
usually not interested in<sup class="footnote-reference"><a href="#2">2</a></sup>. Recently, <a href="https://github.com/ggreer/the_silver_searcher">ag</a> aka The Silver Searcher by Geoff
Greer made the rounds. It is much faster than ack but provides an ack-compatible
interface, hence with <a href="https://github.com/mileszs/ack.vim">ack.vim</a> and</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">let </span><span style="color:#5597d6;">g:ackprg</span><span>=</span><span style="color:#d07711;">"ag --nogroup --nocolor --column"
</span></code></pre>
<p>you have ag at your finger tips.</p>
<p>Because ack was not extremely fast, I didn’t use it that often and never felt
bothered calling it via <code>:Ack</code> and search for the current word with <code><C-r><C-w></code>
from the command line. Now, it bothers me because launching the search should
not be slower than the actual search. Hence I mapped the <code>:Ack</code> command with</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">nnoremap </span><span style="color:#a2a001;"><C-i></span><span> :Ack! </span><span style="color:#a2a001;"><C-r><C-w><CR>
</span></code></pre>
<p>Now, it’s a breeze to search for terms.</p>
<h2 id="folding">Folding</h2>
<p>I’ve written about how I embraced folding <a href="/2013/08/22/vim-folding-for-tex.html">already</a>. I mark all important dot
files with fold markers and try to use them in all source files wherever I can.
For that purpose, I use a fork of <a href="https://github.com/matze/vim-markdown-folding">vim-markdown-folding</a> and my own
<a href="https://github.com/matze/vim-ini-fold">vim-ini-fold</a> and <a href="https://github.com/matze/vim-tex-fold">vim-tex-fold</a> plugins. One big productivity boost comes
from mapping <code><CR></code> to <code>za</code> which opens and closes a fold.</p>
<p>By default, the fold text (the compressed representation of the folded text)
does not look very appealing. But it can be changed pretty easily with a custom
fold text function like <a href="https://github.com/matze/configs/blob/82104a2529307d1ef17e6c79c520641d81129eb9/.vimrc#L195">this</a>.</p>
<h2 id="one-word-about-configuration-management">One word about configuration management</h2>
<p>Maintaining the old shell script for linking dot files and setting up Git
submodules was working but not in the true spirit of “Don’ Repeat Yourself”
because I had to edit it for each new dot file that I included.</p>
<p>Luckily, I stumbled upon <a href="https://github.com/staticshock/seashell">seashell</a> that takes care of such things through a
sole <code>Makefile</code>. By default, <code>make</code> will initialize Git submodules found in the
dotfile repository and <code>make install</code> will create symbolic links for any file or
directory starting with a dot except for Git-specific files. Also, any files
that ends with <code>.export</code> will be linked to without the <code>.export</code> suffix. Hence,
you can have <code>.gitconfig</code> ignoring files in your dotfile repository and
<code>.gitconfig.export</code> ignoring files in all Git repositories.</p>
<p>An additional <code>Makefile.conf</code> provides hooks to customize the initialization and
export steps. This comes in handy for programs that store their configuration
files in <code>$XDG_CONFIG_DIR</code>, e.g. <code>~/.config</code>. Instead of versioning the entire
<code>~/.config</code> dir, you can use</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#5597d6;">EXPORT_CONTENT </span><span style="color:#72ab00;">+= </span><span style="color:#d07711;">.config
</span></code></pre>
<p>to link everything <em>inside</em> <code>$DOT_DIR/.config</code> to <code>~/.config</code>. Another nice
feature of seashell is post-installation code, that can be defined in the
configuration. For example, msmtp requires that its configuration file can only
be accessed by the user, hence adding</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#72ab00;">define</span><span style="color:#5597d6;"> install_append
</span><span style="color:#d07711;">@chmod 600 $(HOME)/.msmtprc
</span><span style="color:#72ab00;">endef
</span></code></pre>
<p>ensures that this is the case.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>The last commits were authored more than a year and committed ten months ago.
<sup class="footnote-reference"><a href="#2">2</a></sup>: For example, ack ignores <code>.git</code> directories and <code>backup~</code>.</p>
</div>
Vim folding for TeX2013-08-22T00:00:00+00:002013-08-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/vim-folding-for-tex/<p>Aeons <a href="https://github.com/matze/vim-tex-fold/commit/29b127439529a42ee45e4f4f6381e1432500a31a">ago</a>, I wrote a small Vim plugin for folding TeX sources. It was a
pathetic attempt, that never worked well and finally sunk into the depths of
forgotten repositories.</p>
<p>Fast-forward to today and you can see me in love with folds. I’ve never been a
big fan of them, but I began to like the idea after I started sprinkling markers
in files which do not have a syntactic hierarchy (e.g. configuration files). To
make a long story short: I revived the <a href="https://github.com/matze/vim-tex-fold">plugin</a> and released a first version
on <a href="http://www.vim.org/scripts/script.php?script_id=4702">vim.org</a>.</p>
<figure>
<img src="/images/2013-08-22/vim-tex-fold.png" />
</figure>
<p>As of now, it works reliably with any kind of section and <code>\begin{}\end{}</code>
environments. I also added regex matches for marker pairs, so you can use them
in the same document as the expression folds. This is useful to group
configuration sections and hide them when not editing them<sup class="footnote-reference"><a href="#1">1</a></sup>. I also added a
<code>foldtext</code> function that removes all the TeX cruft and adds little Unicode icons
in front of the particular fold text.</p>
<p>Again, if you have any suggestions or problems, don’t hesitate to open an issue
on the [issue tracker][].</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Which happens most of the time. And which required skipping large swaths of text, <em>everytime</em> I open a TeX file.
[issue tracker]: https://github.com/matze/vim-tex-fold/issues</p>
</div>
Comments enabled2013-08-17T00:00:00+00:002013-08-17T00:00:00+00:00
Unknown
https://bloerg.net/posts/comments-enabled/<p>Since the inception of this blog, I hesitated to add comments because I didn’t
want to clutter it with unnecessary JavaScript and rely on a third-party service
such as Disqus. Instead, I sometimes link to the corresponding post announcement
on Google+. This has two disadvantages: First, people who are not on Google+
cannot participate in a discussion and second, all comments are kept at Google
which does not make it any better than hosting them at Disqus.</p>
<p>Of course, I like getting feedback and discuss about various issues raised in my
post, thus effectively as of now, you can comment here with a static comment
system.</p>
<h2 id="static-comments">Static comments</h2>
<p>Some time ago, <a href="http://theshed.hezmatt.org/jekyll-static-comments/">Matt Palmer</a> proposed a mail-based system that uses a small
server-side script that sends him mails of new comments in YAML format and a
small Jekyll plugin that reads the comments and makes them available in the
templates. I was intrigued by the idea but because my MUA
<a href="http://www.mozilla.org/thunderbird/">sucks</a>, I had to find another way to
manage incoming comments.</p>
<p>First, I wrote a Flask-based app that receives the POST data from the comment
form, validates the input, computes the Gravatar ID from the mail address and
stores the result in a <a href="https://github.com/mpalmer/jekyll-static-comments/blob/master/static_comments.rb">static_comments.rb</a>-compatible file. I will then pull
the comments via <code>rsync</code> whenever I can devote the time, remove spam and
re-generate the site if necessary.</p>
<h2 id="privacy-and-filtering">Privacy and filtering</h2>
<p>I am not yet ready to publish the source for the server-side app but I can tell
you what I am (not) doing with your data:</p>
<ol>
<li>I use your IP address to limit submissions to one comment per minute. I <em>do
not</em> store your IP address.</li>
<li>I <em>do not</em> publish nor store your mail address. Its only purpose is to
compute a Gravatar ID, so in general I am fine if you put in some bogus
address.</li>
<li>I <em>do</em> process your comments: I strip out tags looking like XML and at some
point the comment gets processed by a Markdown converter. So, what is shown
on the site might not necessarily be exactly the same as what you put in the
box.</li>
<li>Furthermore, I <em>will</em> delete comments with threatening, harassing, racist,
sexist and copyright infringing content.</li>
<li>I will <em>not</em> delete or modify already published comments.</li>
</ol>
Shift lines and selections faster with vim-move2013-08-10T00:00:00+00:002013-08-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/shift-lines-and-selections-faster-with-vim-move/<p>One of my <a href="https://github.com/Dynalon">students</a> showed me a cool <a href="http://www.sublimetext.com/">Sublime Text</a> feature that made
editing a bit more efficient than Vim: Instead of moving lines by deleting them
with <kbd>d</kbd><kbd>d</kbd>, jumping to the desired location and pasting them,
he just had to press a key combination. Because it looked so intuitive, I had to
figure out how to emulate that with Vim.</p>
<p>First thing I stumbled upon and never knew before, is the <code>:m[ove]</code> <a href="http://vimdoc.sourceforge.net/htmldoc/change.html#:move">command</a>.
It takes a range (or the current line) and moves the lines in that range to some
target “address”. The simplest possible movement is the <code>:m+1</code> shortcut, that
swaps the current line with the one right below. Typing the command manually is
a chore but <a href="http://vim.wikia.com/wiki/Moving_lines_up_or_down#Mappings_to_move_lines">mapping</a> that command and its inverse <code>:m-2</code> is pretty
straightforward once you take care of the top and bottom boundaries. Moving
whole selections up and down is more interesting. Fortunately, you can get
easily retrieve selected range in a function and adaptively call <code>:m</code>.</p>
<p>To make these mappings more manageable, I wrote a simple but pretty effective
plugin called <a href="https://github.com/matze/vim-move">vim-move</a>. Put the <code>matze/vim-move</code> repo into your <a href="https://github.com/gmarik/vundle">Vundle</a>
bundle and this is what you get:</p>
<figure><img src="http://i.imgur.com/RMv8KsJ.gif"/></figure>
<p>By default, <kbd>alt</kbd>+<kbd>k</kbd> and <kbd>alt</kbd>+<kbd>j</kbd> move the
current line in normal mode<sup class="footnote-reference"><a href="#1">1</a></sup> whereas <kbd>ctrl</kbd>+<kbd>k</kbd> and
<kbd>ctrl</kbd>+<kbd>j</kbd> shift a selection in visual mode. You can prefix
the calls with a count, e.g. <kbd>5</kbd><kbd>ctrl</kbd>+<kbd>j</kbd> which will
then shift the line or selection relative by that number. To remap the keys you
should</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">let </span><span style="color:#5597d6;">g:move_map_keys</span><span> = </span><span style="color:#b3933a;">0
</span></code></pre>
<p>and then map the keys, for example</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">vmap </span><span style="color:#a2a001;"><C-j> <Plug></span><span>MoveBlockDown
</span><span style="color:#b39f04;">vmap </span><span style="color:#a2a001;"><C-k> <Plug></span><span>MoveBlockUp
</span><span style="color:#b39f04;">nmap </span><span style="color:#a2a001;"><A-j> <Plug></span><span>MoveLineDown
</span><span style="color:#b39f04;">nmap </span><span style="color:#a2a001;"><A-k> <Plug></span><span>MoveLineUp
</span></code></pre>
<p>Hope you like the plugin. If you have any suggestions or problems, open an issue
on the <a href="https://github.com/matze/vim-move/issues">issue tracker</a>.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>In some environments I have to use the GNOME terminal and need to re-map the normal mode keys because I am not able to stop the terminal from intercepting the <kbd>alt</kbd> key. If you know how to let the GNOME terminal use the <kbd>alt</kbd> key, I’d pay you a beer. <em>Update</em>: you can keep your wallets closed ;-)</p>
</div>
pkgconfig for human beings2013-08-08T00:00:00+00:002013-08-08T00:00:00+00:00
Unknown
https://bloerg.net/posts/pkgconfig-for-human-beings/<p>Today, I had to scratch an itch that concerned calling <a href="http://www.freedesktop.org/wiki/Software/pkg-config/">pkg-config</a> from
within Python. Imagine that you write an extension module that depends on some
external development libraries. First you want to make sure that you build
against a version that provides the features you expect and second you have to
provide pre-processor directives, compiler flags and library options to build
your module. The <code>pkg-config</code> command line tool can be used to query this meta
information from its database, at least that’s how it’s done in the manual
Makefile and Autotools world. To gather that information in an arbitrary Python
script, I wrote a small helper module called <a href="https://github.com/matze/pkgconfig">pkgconfig</a>.</p>
<p>To fill this empty post with more blurb, I will restate the README here:
<code>pkgconfig</code> can be used to check if a package exists</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">>>> import </span><span>pkgconfig
</span><span style="color:#72ab00;">>>> </span><span>pkgconfig.</span><span style="color:#5597d6;">exists</span><span>(</span><span style="color:#d07711;">'glib-2.0'</span><span>)
</span><span style="color:#b3933a;">True
</span></code></pre>
<p>check if a package meets certain version requirements</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span>
</span><span style="color:#72ab00;">>>> </span><span>pkgconfig.</span><span style="color:#5597d6;">installed</span><span>(</span><span style="color:#d07711;">'glib-2.0'</span><span>, </span><span style="color:#d07711;">'< 2.26'</span><span>)
</span><span style="color:#b3933a;">False
</span><span style="color:#72ab00;">>>> </span><span>pkgconfig.</span><span style="color:#5597d6;">installed</span><span>(</span><span style="color:#d07711;">'glib-2.0'</span><span>, </span><span style="color:#d07711;">'>=2.26'</span><span>)
</span><span style="color:#b3933a;">True
</span></code></pre>
<p>and also query the <code>CFLAGS</code> and <code>LDFLAGS</code> for building a dependent module</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">>>> </span><span>pkgconfig.</span><span style="color:#5597d6;">cflags</span><span>(</span><span style="color:#d07711;">'glib-2.0'</span><span>)
</span><span style="color:#d07711;">'-I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include'
</span><span>
</span><span style="color:#72ab00;">>>> </span><span>pkgconfig.</span><span style="color:#5597d6;">libs</span><span>(</span><span style="color:#d07711;">'glib-2.0'</span><span>)
</span><span style="color:#d07711;">'-lglib-2.0'
</span></code></pre>
<p>The module is licensed under the MIT license, so if you want to use it in your
private projects, just dump it there. Suggestions and bug reports
should be addressed in the GitHub <a href="https://github.com/matze/pkgconfig">repo</a>.</p>
<p>PS: I know, it’s the worst project tag line in the history of project tag lines.</p>
Static photo galleries2013-07-30T00:00:00+00:002013-07-30T00:00:00+00:00
Unknown
https://bloerg.net/posts/static-photo-galleries/<p>Recently, I purchased a Canon 40mm <em>f</em>/2.8 STM lens for my trusty Canon 550D.
Now that I shoot more often again, I became worried with my habit to store
images on Google+. Nothing against Google+’s photo section: it features an
incredibly good-looking UI, simple upload and tight integration with the rest of
Google’s services. Moreover, I don’t have to worry about traffic and
availability. Nevertheless, it is in the cloud and comes with all those cloudy
strings attached. Hence, I set out to find an alternative that I could host on
my own. The alternative had to be sufficiently simple to setup but provide
enough customization options and “hackability” for my needs.</p>
<p>Soon enough, I found out about the <a href="http://sigal.saimon.org/en/latest/">Sigal</a>
project by Simon Conseil. It’s very similar to static site generator for blogs,
but instead of transforming Markdown into HTML, it crawls through a source
directory and generates album and gallery pages. Out of the box, it comes with
two pre-defined themes based on the
<a href="http://www.jacklmoore.com/colorbox">Colorbox</a> and
<a href="http://galleria.io/">Galleria</a> libraries. However, I <em>had</em> to customize it. I
quickly found out that the themes are based on <a href="http://sass-lang.com/">SASS</a>
which the Sigal compiler cannot turn into CSS by itself. The easiest way to
achieve this, was to integrate the conversion step as a make target in a
Makefile that I use to control the build process. To keep everything
self-contained, I use pyScss instead of the Ruby-based original. So, to make a
long story short, here is the Makefile to build and deploy the gallery site:</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#5597d6;">LOAD_PATH</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">_theme/static/css
</span><span style="color:#5597d6;">SCSS</span><span style="color:#72ab00;">=$(</span><span style="color:#acb3c2;">LOAD_PATH</span><span style="color:#72ab00;">)</span><span style="color:#d07711;">/style.scss
</span><span style="color:#5597d6;">CSS</span><span style="color:#72ab00;">=$(</span><span style="color:#acb3c2;">LOAD_PATH</span><span style="color:#72ab00;">)</span><span style="color:#d07711;">/style.min.css
</span><span>
</span><span style="color:#c23f31;">.PHONY</span><span style="color:#72ab00;">: </span><span style="color:#d07711;">all build rebuild serve deploy
</span><span>
</span><span style="color:#c23f31;">all</span><span style="color:#72ab00;">: </span><span style="color:#d07711;">build
</span><span>
</span><span style="color:#c23f31;">serve</span><span style="color:#72ab00;">: </span><span style="color:#d07711;">build
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">sigal</span><span> serve
</span><span>
</span><span style="color:#c23f31;">build</span><span style="color:#72ab00;">: $(</span><span style="color:#acb3c2;">CSS</span><span style="color:#72ab00;">)
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">sigal</span><span> build
</span><span>
</span><span style="color:#c23f31;">rebuild</span><span style="color:#72ab00;">:
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">sigal</span><span> build</span><span style="color:#5597d6;"> -f
</span><span>
</span><span style="color:#72ab00;">$(</span><span style="color:#5597d6;">CSS</span><span style="color:#72ab00;">): $(</span><span style="color:#acb3c2;">SCSS</span><span style="color:#72ab00;">)
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">python -mscss -I </span><span style="color:#72ab00;">$(</span><span style="color:#5597d6;">LOAD_PATH</span><span style="color:#72ab00;">) < </span><span style="color:#5597d6;">$< </span><span style="color:#72ab00;">> </span><span style="color:#5597d6;">$@
</span><span>
</span><span style="color:#c23f31;">deploy</span><span style="color:#72ab00;">: </span><span style="color:#d07711;">build
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">rsync -r -c -z</span><span> _build/ foo@bar.com:/path/to/www/data
</span></code></pre>
<p>Whenever I put something on <del>Picasa</del> Google+, not much later someone
asked “how can I download <em>all</em> the pictures” and I had to respond that it’s
just not possible. Right out of the box, this is also not possible with Sigal.
But in the end, it was just a couple of lines of code to add that feature.
Another feature, that I really wanted to have in my galleries are EXIF data, you
know, focal length, aperture, exposure time, ISO setting etc. Both changes are
waiting to be merged into the main repository of Simon.</p>
<p>With these changes, you can get something like this (the design is not yet
finalized though):</p>
<figure>
<img src="/images/2013-07-30/gallery.jpg" />
</figure>
<p><em>Update</em>: Both pull requests landed and it <em>is</em> possible to download Zip
archives and display the EXIF tags. Thanks Simon.</p>
OpenCL 2.0 is here2013-07-22T00:00:00+00:002013-07-22T00:00:00+00:00
Unknown
https://bloerg.net/posts/opencl-2-0-is-here/<p>Today at SIGGRAPH ’13, Khronos <a href="https://www.khronos.org/news/press/khronos-releases-opencl-2.0">released</a> a provisional specification
specification of OpenCL 2.0. Hooray! Compared to 1.2, this is release is packed
with unexpected yet cool stuff and features that were requested by users for
quite some time now. The provisional specification can be found <a href="http://www.khronos.org/opencl/">online</a>,
however the manpages are not fully built yet.</p>
<p>So, let’s don’t waste time. Here is what’s coming up.</p>
<h2 id="kernel-argument-introspection">Kernel argument introspection</h2>
<p>I can only imagine that the Khronos developers came up with the existing
<code>clSetKernelArg</code> API, to provide programmatic access from languages other than
C. However even with these good intents, it was not even possible to determine
the types of the argument of a <code>clKernel</code> object hence making it extremely
difficult to provide a better API in C itself. After many users requested a way
to introspect kernel arguments, OpenCL 2.0 now brings <a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/clGetKernelArgInfo.html">clGetKernelArgInfo</a>.
Unfortunately, it still is a bit awkward to use. For example, you cannot get the
size in bytes of the argument but instead you can query for the type name, from
which you <em>could</em> infer this information.</p>
<h2 id="shared-virtual-memory">Shared Virtual Memory</h2>
<p>Until version 1.2, you could use the same memory from host and device by using
the <code>clEnqueueMapBuffer</code> and <code>clEnqueuUnmapBuffer</code> operations. However, this was
restricted to contiguous memory buffers. OpenCL 2.0 provides a slew of
<a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/sharedVirtualMemory.html">functions</a> dedicated for <em>shared virtual memory</em> (… I guess pages) that
allow access of arbitrary pointer-based data structures.</p>
<h2 id="pipes">Pipes</h2>
<p>I am not sure about the usefulness of the new pipe objects. Apparently, you
<a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/clCreatePipe.html">create</a> a new pipe on the host and pass the <code>cl_mem</code> object to two kernels
that use a number of new built-in <a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/sharedVirtualMemory.html">functions</a> to communicate through these
objects. As far as I can see, the pipes only allow buffered, asynchronous
operation.</p>
<p>At the moment, I can hardly see any use case for these new objects. Do you?</p>
<h2 id="enqueueing-kernels-from-within-kernels">Enqueueing kernels from within kernels</h2>
<p>… also called “Dynamic Parallelism” by the Khronos folks. The new built-in
<a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/enqueue_kernel.html">enqueue_kernel</a> allows a work item to schedule a new kernel. Instead of a
receiving a kernel object or function pointer to a <code>kernel</code> function, the
consortium decided to re-use Apple’s <em>blocks</em> to create enqueue-able
closures. According to the specification, enqueueing looks like this:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span>kernel </span><span style="color:#668f14;">void
</span><span style="color:#c23f31;">outer </span><span>(global </span><span style="color:#668f14;">int </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">a</span><span>, global </span><span style="color:#668f14;">int </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">b</span><span>)
</span><span>{
</span><span> ndrange_t range;
</span><span>
</span><span> </span><span style="color:#668f14;">void </span><span>(</span><span style="color:#72ab00;">^</span><span>accumulate)(</span><span style="color:#668f14;">void</span><span>) </span><span style="color:#72ab00;">=
</span><span> </span><span style="color:#72ab00;">^</span><span>{
</span><span> </span><span style="color:#a2a001;">size_t</span><span> id </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">get_global_id </span><span>(</span><span style="color:#b3933a;">0</span><span>);
</span><span> b[id] </span><span style="color:#72ab00;">+=</span><span> a[id];
</span><span> };
</span><span>
</span><span> </span><span style="color:#7f8989;">/* Pass block as variable */
</span><span> </span><span style="color:#5597d6;">enqueue_kernel </span><span>(</span><span style="color:#5597d6;">get_default_queue </span><span>(),
</span><span> CLK_ENQUEUE_FLAGS_WAIT_KERNEL,
</span><span> range,
</span><span> accumulate);
</span><span>
</span><span> </span><span style="color:#7f8989;">/* Pass block directly */
</span><span> </span><span style="color:#5597d6;">enqueue_kernel </span><span>(</span><span style="color:#5597d6;">get_default_queue </span><span>()
</span><span> CLK_ENQUEUE_FLAGS_WAIT_KERNEL,
</span><span> range,
</span><span> </span><span style="color:#72ab00;">^</span><span>{
</span><span> </span><span style="color:#a2a001;">size_t</span><span> id </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">get_global_id </span><span>(</span><span style="color:#b3933a;">0</span><span>);
</span><span> b[id] </span><span style="color:#72ab00;">= </span><span style="color:#b39f04;">sin</span><span>(a[id]);
</span><span> });
</span><span>}
</span></code></pre>
<h2 id="spir-intermediate-representation">SPIR intermediate representation</h2>
<p>This is probably less of importance for users of the C and C++ API, but the
Khronos group also started to define a LLVM-based intermediate representation of
up to OpenCL 1.2 code. This will probably spawn a multitude of new OpenCL
front-end languages.</p>
<h2 id="other-built-ins-and-extensions">Other built-ins and extensions</h2>
<ul>
<li>Tired of calculating the linear index for two- and three-dimensional data
accesses? With the upcoming specification you can use <a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/get_global_linear_id.html">get_global_linear_id</a>
and <a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/get_local_linear_id.html">get_local_linear_id</a> for these tasks.</li>
<li>There is now a whole <a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/atomicFunctions.html">bunch</a> of atomic functions to change memory locations
or test for conditions while being immediately visible to other work items of
the same work group.</li>
<li>Sometimes, it’s a pain to get resource de-allocation <a href="/2013/01/15/opencl-resource-management.html">right</a>. For sloppy
people like me, there is a new <a href="http://www.khronos.org/registry/cl/sdk/2.0/docs/man/xhtml/cl_khr_terminate_context.html">cl_khr_terminate_context</a> extension, that
provides a <code>clTerminateContextKHR</code> to quickly shut down an OpenCL context.</li>
</ul>
<h2 id="outlook">Outlook</h2>
<p>The specification is still in provisional state and the Khronos group encourages
you to give feedback for the next six months. After that it will be set in
stone. Probably two years later, NVIDIA will then provide an implementation for
OpenCL 1.2.</p>
AMD's OpenCL multi GPU bug2013-07-10T00:00:00+00:002013-07-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/amds-opencl-multi-gpu-bug/<p>Implementing open standards such as OpenCL seems to be a daunting and <a href="/2012/07/19/heterogenous-computing.html">error
prone</a> task. Here’s a problem that went unnoticed for quite some time on an
AMD-based system.</p>
<p>To compile an OpenCL kernel for several devices, one has to use <code>clBuildProgram</code>
that takes a program created with <code>clCreateProgramWithSource</code> or
<code>clCreateProgramWithBinary</code> and a list of devices. In plain C it looks like
this:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#668f14;">static const char </span><span style="color:#72ab00;">*</span><span>source </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">"__kernel void foo(void) {}"</span><span>;
</span><span>cl_program program;
</span><span>cl_device_id devices[</span><span style="color:#b3933a;">2</span><span>];
</span><span>
</span><span>program </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">clCreateProgramWithSource </span><span>(context, </span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#72ab00;">&</span><span>source, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">NULL</span><span>);
</span><span style="color:#5597d6;">clBuildProgram </span><span>(program, </span><span style="color:#b3933a;">2</span><span>, devices, </span><span style="color:#d07711;">""</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">NULL</span><span>);
</span></code></pre>
<p>In a heterogeneous environment, it is possible that different devices of the
same vendor happily co-exist on the same platform. At least, that’s the core
idea of OpenCL. In order to write optimized kernel code, we can define
pre-processor defines and compile the kernel for each device, just like this:</p>
<pre data-lang="c" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-c "><code class="language-c" data-lang="c"><span style="color:#5597d6;">clBuildProgram </span><span>(program, </span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#72ab00;">&</span><span>devices[</span><span style="color:#b3933a;">0</span><span>], </span><span style="color:#d07711;">"-DTYPE=FOO"</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">NULL</span><span>);
</span><span style="color:#5597d6;">clBuildProgram </span><span>(program, </span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#72ab00;">&</span><span>devices[</span><span style="color:#b3933a;">1</span><span>], </span><span style="color:#d07711;">"-DTYPE=BAR"</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">NULL</span><span>);
</span></code></pre>
<p>Inside the kernel you could write conditional code with <code>#ifdef</code> and switch
between two alternative implementations. However, this works only on NVIDIA
hardware. Once you enqueue a kernel on the first command queue (the one attached
to the first device) of an AMD device, you will receive a
<code>CL_INVALID_PROGRAM_EXECUTABLE</code> error. In my humble opinion this is a clear
violation against the spec:</p>
<blockquote>
<p>The function [clBuildProgram] builds (compiles & links) a program executable
from the program source or binary for all the devices or a specific device(s)
in the OpenCL context associated with <em>program</em>.</p>
</blockquote>
<p>When I interpret this correctly, the standard does not restrict usage on
“specific device(s)”, hence I must be able to run <code>clBuildProgram</code> as often as I
wish. Now, to check if you are affected, you can use my <a href="https://github.com/matze/ocl-regressions">regression tool</a>.
When run on an AMD system, you will see something like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span># Platform: OpenCL 1.2 AMD-APP (1084.4)
</span><span># Device 0: Tahiti
</span><span># Device 1: Tahiti
</span><span># Device 2: Tahiti
</span><span># Device 3: Tahiti
</span><span>[...]
</span><span>Creating program `amd_bug`: OK
</span><span>Build program for both GPU 1: OK
</span><span>Build program for both GPU 2: OK
</span><span>Created kernel `amd_bug': OK
</span><span>Started kernel `amd_bug' on queue 0 [expect CL_INVALID_PROGRAM_EXECUTABLE]: Error: CL_INVALID_PROGRAM_EXECUTABLE
</span><span>Started kernel `amd_bug' on queue 1 [expect CL_INVALID_PROGRAM_EXECUTABLE]: OK
</span></code></pre>
<p>As with NVIDIA’s bug, there is no real workaround for this issue. So, let’s hope
that AMD will address this problem.</p>
Locking up2013-06-29T00:00:00+00:002013-06-29T00:00:00+00:00
Unknown
https://bloerg.net/posts/locking-up/<p>In the light of recent events, I decided to take a little bit more care of my
privacy. I am not overly paranoid concerning my data but it seems I need a
certain level of confidence how the data can or cannot be accessed by third
parties.</p>
<h2 id="sign-and-encrypt-mails-with-gpg">Sign and encrypt mails with GPG</h2>
<p>I created a new GPG key pair according to <a href="http://alexcabal.com/creating-the-perfect-gpg-keypair/">this</a> guide. For convenience
reasons, I also removed the master signing key from my keyring. Thus when a
malicious thief gets hold of it, I can revoke all signed subkeys but keep the
accumulated trust in my master key.</p>
<p>This is the fingerprint of my currently used 4096 bit long <a href="http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x018A14794D3A8106">key pair</a>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>DD89 68B2 F64F 0260 A387 5EE0 018A 1479 4D3A 8106
</span></code></pre>
<p>All other keys that I used in the past are invalid from now on.</p>
<p>Also, I try to avoid the GMail web interface as much as possible and use
Thunderbird together with the excellent Enigmail plugin instead. All out-going
mails are signed and encrypted if possible.</p>
<h2 id="leaving-the-cloud">Leaving the cloud</h2>
<p>I removed all sensitive data from Dropbox and Google Drive. I know, they
probably backed it up anyway but it’s a start. The most sensitive data (GPG
master key, scanned documents, etc.) are saved on a USB thumb drive in an EncFS
directory. It is the easiest and most comfortable solution I encountered so far.
This should get you started:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ sudo apt-get install encfs
</span><span>$ encfs /path/to/usb/.secret /path/to/usb/secret
</span><span>...
</span><span>$ fusermount -u /path/to/usb/secret
</span></code></pre>
<p>I wrote a little shell script saving me from typing this over and over again.
If the <code>secret</code> directory is empty, it runs the <code>encfs</code> line otherwise it
unmounts it.</p>
<h2 id="conclusion">Conclusion</h2>
<p>As said before, I am not a paranoid guy and I will try to be as open and
<em>public</em> as possible as long as it does not touch my personal privacy.</p>
Use rubber to compile TeX2013-06-25T00:00:00+00:002013-06-25T00:00:00+00:00
Unknown
https://bloerg.net/posts/use-rubber-to-compile-tex/<p>Compiling TeX documents by hand is tedious<sup class="footnote-reference"><a href="#1">1</a></sup> and with all the output a pain to
watch at. For a long time, I used [latex-makefile][] that tries to solve both
problems. However, it is a huge beast which broke compilation from time to time.</p>
<p>After looking around a bit, I came to [latexmk][] which <em>almost</em> fit the bill.
The Perl script does what it has to do but [requires][] patching latexmk itself
or a <code>.latexmkrc</code> file just for compiling XeLaTeX sources.</p>
<p>So, I looked a little further and found [rubber][]. I knew it already from
Gedit’s LaTeX plugin but I wasn’t really aware that one could use it as a
standalone solution. Anyway, for simple projects calling</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#5597d6;">$</span><span> rubber</span><span style="color:#5597d6;"> --pdf</span><span> doc
</span></code></pre>
<p>compiles <code>doc</code> with pdfLaTeX, strips out unnecessary debug output and
pre-processes things like the BibTeX bibliography. To compile XeLaTeX sources,
you could [either][] write a rubber module and distribute it with your document,
or simply add</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>% rubber: set program xelatex
</span></code></pre>
<p>on top of your source file. That’s it.</p>
<p>Now, instead of the huge latex-makefile or a hand-written Makefile, I use this
little baby for all my TeX documents:</p>
<pre data-lang="make" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-make "><code class="language-make" data-lang="make"><span style="color:#5597d6;">SRC</span><span style="color:#72ab00;">=$(</span><span style="color:#b39f04;">wildcard </span><span style="color:#acb3c2;">*</span><span style="color:#d07711;">.tex</span><span style="color:#72ab00;">)
</span><span style="color:#5597d6;">OPTS</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">--pdf
</span><span>
</span><span style="color:#c23f31;">.PHONY</span><span style="color:#72ab00;">: </span><span style="color:#d07711;">clean
</span><span>
</span><span style="color:#c23f31;">all</span><span style="color:#72ab00;">: $(</span><span style="color:#b39f04;">patsubst </span><span style="color:#acb3c2;">%</span><span style="color:#d07711;">.tex,</span><span style="color:#acb3c2;">%</span><span style="color:#d07711;">.pdf,</span><span style="color:#72ab00;">$(</span><span style="color:#acb3c2;">SRC</span><span style="color:#72ab00;">))
</span><span>
</span><span style="color:#5597d6;">%</span><span style="color:#c23f31;">.pdf</span><span style="color:#72ab00;">: </span><span style="color:#acb3c2;">%</span><span style="color:#d07711;">.tex
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">rubber </span><span style="color:#72ab00;">$(</span><span style="color:#5597d6;">OPTS</span><span style="color:#72ab00;">) </span><span style="color:#5597d6;">$<
</span><span>
</span><span style="color:#c23f31;">clean</span><span style="color:#72ab00;">:
</span><span> </span><span style="color:#b3933a;">@</span><span style="color:#5597d6;">rubber --clean </span><span style="color:#72ab00;">$(</span><span style="color:#5597d6;">OPTS</span><span style="color:#72ab00;">) $(</span><span style="color:#5597d6;">SRC</span><span style="color:#72ab00;">)
</span></code></pre>
<p>More information about rubber can be found in an [entry][] of the TeX SE
community blog.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p><code>pdflatex && bibtex && pdflatex && pdflatex</code> … seriously?
[latex-makefile]: http://code.google.com/p/latex-makefile/
[latexmk]: http://www.ctan.org/pkg/latexmk/
[rubber]: https://launchpad.net/rubber/
[requires]: http://stackoverflow.com/questions/3124273/compile-xelatex-tex-file-with-latexmk
[entry]: http://tex.blogoverflow.com/2011/12/building-documents-with-rubber/
[either]: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=579757</p>
</div>
HacKAthon 2013 recap2013-06-12T00:00:00+00:002013-06-12T00:00:00+00:00
Unknown
https://bloerg.net/posts/hackathon-2013-recap/<p>Last weekend a friend and I attended this years HacKAthon organized by
<a href="http://www.nerd-zone.com">nerd-zone</a>. This was my very first hackathon and as such I didn’t really knew
what to expect or how much to get out of it.</p>
<p>After a short “pitch” (serious people seriously take that serious!) on Friday
evening, <a href="https://twitter.com/daniel_aus_wa">Daniel</a> and <a href="http://xam.de">Max</a> displayed interest for a collaboration. After
some back and forth, we halfway completed a web application that helps getting
to know people in a restricted spatial and temporal environment by visualizing
common interests. For this, everyone is tagged by anyone and the larger the conjuctive subset
between two persons, the fatter the connecting edge.</p>
<figure>
<img src="/images/2013-06-12/gravitee.png" />
</figure>
<p>A real-time update mechanism helps visualizing the login and tagging processes.
If you are interested, you can have a look at a <a href="http://frozen-chamber-7393.herokuapp.com/">live demo</a> and the source on
<a href="https://github.com/matze/hackathon2013">GitHub</a>. For the demo, we used Python, JavaScript, CSS3 and HTML5. The
backend was realised with Django, the front end is a combination of
<a href="http://knockoutjs.com">Knockout.js</a> and <a href="http://cytoscape.github.io/cytoscape.js/">Cytoscape.js</a> to visualize the network.
Last but not least, we deployed the app on <a href="http://heroku.com">Heroku</a>.</p>
<h2 id="what-was-good">What was good</h2>
<p>Getting to know new people with a different background than myself, was probably
the most interesting part of the hackathon. Moreover, hacking on things that
is usually not part of my daily work has some positive side effects.</p>
<p>Also, working focused in a team on a particular project in a very short amount
of time is an interesting experience. Maybe, the same is happening in a
corporate environment but I doubt it generates as much positive stress as it
does on a hackathon.</p>
<p>From an organization point of view, most things went smooth: The wireless was
fast and without hiccups, but that should actually be no big deal for one of
Germany’s larger <a href="http://www.1und1.de">ISPs</a>. Also, there was an endless stream of free caffeinated
beverages and stereotypical but good food (pizza & schnitzel).</p>
<h2 id="what-could-ve-been-better">What could’ve been better</h2>
<p>The topic of the hackathon itself was clearly stated but not restricted in any
meaningful way. It was so broad that the final decision who won was a bit
arbitrary. Nevertheless, it also brought in a huge
<a href="http://bit.ly/hacKAthon2013">variety</a> of apps.</p>
<p>Althought it didn’t affect us, the hackathon itself was not a true 24 hour
coding marathon but the place was locked from 1.30 am until 8.30 am because 1&1
missed to prolong the security guard’s duty. This should’ve been organized
better because there were people willing to pull an all-nighter.</p>
<h2 id="my-conclusion">My conclusion</h2>
<p>Overall, it was fun and I would definately attend such an event again.</p>
Tracking GObjects2013-06-06T00:00:00+00:002013-06-06T00:00:00+00:00
Unknown
https://bloerg.net/posts/tracking-gobjects/<p>I stated <a href="/2013/01/15/opencl-resource-management.html">before</a>, that you have to release <em>all</em> OpenCL resources to prevent
leaking buffer memory and eventually crashing the application. Now, it hit me
again but wrapped in a GObject.</p>
<p>I didn’t know where exactly the problem was located but I suspected that a
GObject wasn’t properly unreferenced and in the end finalized, hence leaking
OpenCL resources. I <em>knew</em> there was a project to keep track of GObject
references but I forgot the name.</p>
<p>Eventually, I found <a href="https://github.com/danni/gobject-list">gobject-list</a> again thanks to my Google super powers. Not
only that, but it also helped debugging the problem in question. Now, what is
gobject-list? It is a small library that is pre-load when running the
application:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ LD_PRELOAD=$PATH_TO/libgobject-list.so ./app
</span><span> ++ Created object 0x1ee3800, UfoConfig
</span><span> ++ Created object 0x1ee3840, UfoPluginManager
</span><span> ++ Created object 0x1ee5810, UfoResources
</span></code></pre>
<p>The library intercepts calls to <code>g_object_new</code>, <code>g_object_ref</code> and
<code>g_object_unref</code> and logs them. By sending <code>SIGUSR1</code> to the application,
alive objects and reference counts can be queried:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ killall -USR1 app
</span><span>Living Objects:
</span><span> - 0x1ee5810, UfoResources: 1 refs
</span><span> - 0x1ee3800, UfoConfig: 3 refs
</span><span> - 0x1ee3840, UfoPluginManager: 1 refs
</span></code></pre>
<p>More details can be found in the <a href="http://blogs.gnome.org/danni/2011/02/17/ld_preload-gobject-lifetime-debugging-tool/">blog post</a> by the original author. Big props to
her!</p>
Moving to Jekyll 1.02013-05-06T00:00:00+00:002013-05-06T00:00:00+00:00
Unknown
https://bloerg.net/posts/moving-to-jekyll-1-0/<p>I was happy to see that Jekyll 1.0 was <a href="http://blog.parkermoore.de/2013/05/06/jekyll-1-dot-0-released/">released</a> today and of course I
was eager to move on. Because subcommands are the latest shit now, the
<code>--no-auto</code> option and the destination directory weren’t working in the Git
<code>post-receive</code> hook. Additionally, the <code>auto: true</code> statement in my
<code>_config.yml</code> caused Jekyll to scream at me. Also, Jekyll told me that it could
not find a <code>matches</code> method in the Markdown converter, which seems to be
<a href="http://jekyllrb.com/docs/plugins#converters">required</a> nowadays. Nevertheless, all these problems are easily fixed.</p>
<p>But I also noticed that syntax highlighting and the <a href="/2013/03/03/typographic-improvements-for-jekyll.html">typogruby enhancements</a>
were silently gone. Problem was, that my <a href="/2013/03/07/using-kramdown-instead-of-maruku.html">Kramdown converter</a> was not used for
the Markdown input, even though the <code>matches</code> method looks for the <code>.md</code>
extension. The easiest fix? Use a dummy Markdown extension in <code>_config.yml</code></p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>markdown_ext: foo
</span></code></pre>
<p>and the default converter will be skipped. Apart from these minor problems, the
Jekyll move went smoothly.</p>
<p><em>Update</em>: <a href="http://navarroj.com/research/">Juan Antonio Navarro Pérez</a> kindly
uploaded a version of the “Kramdown + Pygments” converter to
<a href="https://github.com/navarroj/krampygs">GitHub</a>.</p>
Beamer's weird column defaults2013-05-05T00:00:00+00:002013-05-05T00:00:00+00:00
Unknown
https://bloerg.net/posts/beamers-weird-column-defaults/<p>From time to time I encounter a problem with LaTeX or Beamer that costs twice as
much time to solve than what I save by using LaTeX. Here is a reminder for me to
fix a particularly strange behaviour with Beamer’s <code>columns</code> environment.</p>
<p>The <code>columns</code> environment is used to split content horizontally in (you
guessed it) several columns. Intuitively, you’d write something like this</p>
<pre data-lang="tex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-tex "><code class="language-tex" data-lang="tex"><span style="color:#72ab00;">\documentclass</span><span>{</span><span style="color:#a2a001;">beamer</span><span>}
</span><span>
</span><span style="color:#72ab00;">\begin</span><span>{</span><span style="color:#5597d6;">document</span><span>}
</span><span style="color:#b39f04;">\frame</span><span>{
</span><span> </span><span style="color:#b39f04;">\frametitle</span><span>{Slide with columns}
</span><span> </span><span style="color:#72ab00;">\begin</span><span>{</span><span style="color:#5597d6;">columns</span><span>}
</span><span> </span><span style="color:#b39f04;">\column</span><span>{.5</span><span style="color:#b39f04;">\textwidth</span><span>}
</span><span> First column
</span><span>
</span><span> </span><span style="color:#b39f04;">\column</span><span>{.5</span><span style="color:#b39f04;">\textwidth</span><span>}
</span><span> Second column
</span><span> </span><span style="color:#72ab00;">\end</span><span>{</span><span style="color:#5597d6;">columns</span><span>}
</span><span>
</span><span> Conclusive text
</span><span>}
</span><span style="color:#72ab00;">\end</span><span>{</span><span style="color:#5597d6;">document</span><span>}
</span></code></pre>
<p>and expect the columns to be aligned nicely with the adjacent text. But for
whatever reason, that’s not the case by default:</p>
<figure>
<img src="/images/2013-05-05/bad-alignment.png" />
</figure>
<p>To fix this, you have to pass the <code>onlytextwidth</code> option to the environment</p>
<pre data-lang="tex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-tex "><code class="language-tex" data-lang="tex"><span> </span><span style="color:#72ab00;">\begin</span><span>{</span><span style="color:#5597d6;">columns</span><span>}[onlytextwidth]
</span></code></pre>
<p>and you are good to go:</p>
<figure>
<img src="/images/2013-05-05/good-alignment.png" />
</figure>
<p>I don’t know if this happens only with TeX Live 2009, but it is certainly
annoying …</p>
A Git cherry pick text user interface2013-04-25T00:00:00+00:002013-04-25T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-git-cherry-pick-text-user-interface/<p>Lately, I’ve been merging more and more commits between branches using <a href="http://git-scm.com/docs/git-cherry-pick">git
cherry-pick</a> instead of a regular merge. Sometimes this is necessary
because “the other guy” messes up his own branch and merging is just not
feasible and sometimes it is useful because only some changes are of particular
interest.</p>
<p>The workflow usually looks like this: First, I checkout the branch where the
merge should happen. Then, I will have a look at the branch using <code>git log</code>
from where I want to pick commits. Eventually, I cherry pick the commits by
specifying their hashes:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>git cherry-pick ae4971b fe0281b
</span></code></pre>
<p>The last step is tedious and error-prone when typing the hashes by hand, so I
wrote a stupid tool called <a href="http://github.com/matze/git-pick-from">pick-from</a> to simplify this. After specifying from
which branch to pick from</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ pick-from other-branch
</span></code></pre>
<p>pick-from shows me a list of commits that exist in the other branch but not
in the current one:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Choose commits to cherry-pick and accept with `q'
</span><span>
</span><span>[ ] 13579ef: A commit message
</span><span>[X] fa45678: Another commit message
</span></code></pre>
<p>After selecting all desired commits with <kbd>Enter</kbd> and accepting with
<kbd>q</kbd>, the commit hashes are passed unconditionally to <code>git cherry-pick</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>[master fa45678] Another commit message
</span><span> Author: Gandalf the Grey [gandalf@ring.com]
</span><span> 1 file changed, 16 insertions(+), 7 deletions(-)
</span></code></pre>
<p>pick-from is written using the TUI toolkit <a href="http://excess.org/urwid/">urwid</a>, which is practically
available for all distributions. If you want to let the tool look more
“officially” integrated, you can also alias it in your <code>.gitconfig</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>[alias]
</span><span>pick-from = !pick-from
</span></code></pre>
<p>Have fun with that.</p>
Managing notes with the Gollum wiki2013-04-20T00:00:00+00:002013-04-20T00:00:00+00:00
Unknown
https://bloerg.net/posts/moving-notes-to-gollum/<p>I already wrote about how I <a href="/2012/09/14/taking-notes-with-markdown-and-pandoc.html">took notes</a> in Markdown, render them with Pandoc
and keep them in sync with my server via Bitpocket. However, there are several
problems with this approach:</p>
<ol>
<li>Wherever I am, I need SSH access to my machine and Bitpocket must be
installed to keep everything up to date.</li>
<li>Even the smallest changes require mundane tasks.</li>
<li>From within a note I cannot reference and jump to other notes.</li>
</ol>
<p>Obviously, the perfect solution to all my problems would be a web-based wiki
with Markdown formatting. A quick search revealed only two serious
Markdown-based wikis: <a href="http://markdoc.org">Markdoc</a> and <a href="https://github.com/gollum/gollum">Gollum</a>.</p>
<p>Markdoc is for wikis what Jekyll is for blogs: A static site generator for a
bunch of Markdown files. The builtin web server is good for viewing the
generated HTML pages but nothing else. Because Markdoc does not solve any of the
mentioned problems, I took a stab at Gollum.</p>
<p>Gollum was founded by GitHub developers and backs all public GitHub wikis. With
this in mind, it is no wonder that each wiki is backed by a Git repository
rather than a data base. Thus editing a page, simply requires a Git commit.
However for quick edits the builtin web server is much faster.</p>
<p>As with most Ruby programs, you can install Gollum with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>gem install gollum
</span></code></pre>
<p>and run it by pointing it to an existing Git repository:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>git init .
</span><span>gollum .
</span></code></pre>
<p>A big advantage compared to my retired setup is a live editing preview</p>
<figure>
<img src="/images/2013-04-20/live-preview.png" />
</figure>
<p>and <a href="http://www.mathjax.org/">MathJax</a> support</p>
<figure>
<img src="/images/2013-04-20/mathjax.png" />
</figure>
<p>However, you should be aware that the Markdown processor eats escape characters
and that you have to enclose inline equations in <code>\\(</code> and <code>\\)</code>.</p>
<p>To get Gollum (semi-privately) online, I run Gollum locally inside a tmux
session and setup nginx as a reverse proxy that forwards outside requests to the
Gollum service. That’s definitely not a perfect setup but it works very well for
me.</p>
<p>One last word: If you want to serve a statically generated site you could either
use Markdoc or <a href="https://github.com/rubyworks/smeagol">Smeagol</a> (way too many puns intended). I hope this will be me
last infrastructural change to my “note taking system” for the foreseeable
future. Keeping fingers crossed!</p>
Automatizing tasks with Fabric2013-04-18T00:00:00+00:002013-04-18T00:00:00+00:00
Unknown
https://bloerg.net/posts/automatizing-tasks-with-fabric/<p>We all know that good programmers are <a href="http://www.codinghorror.com/blog/2005/08/how-to-be-lazy-dumb-and-successful.html">lazy</a>. <a href="http://fabfile.org">Fabric</a> is another trick that
you should have up your sleeve, if you want to be good:</p>
<blockquote>
<p>Fabric is a Python (2.5 or higher) library and command-line tool for
streamlining the use of SSH for application deployment or systems
administration tasks.</p>
</blockquote>
<p>Although Fabric is often mentioned in web development contexts for deploying
server software to remote machines, it is general enough for all kinds of remote
administration tasks. To give you an analogy: Fabric is for administrative tasks
as to what a Makefile is for build tasks.</p>
<h2 id="defining-fabric-tasks">Defining Fabric tasks</h2>
<p>Fabric tasks are defined in file called <code>fabfile.py</code>, which is read by the
<a href="http://docs.fabfile.org/en/1.6/usage/fab.html">fab</a> command-line tool. Each function defined in this file is accessible from
the command line as a <code>fab</code> parameter. So, if you have something like</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">hello</span><span>():
</span><span> </span><span style="color:#b39f04;">print</span><span>(</span><span style="color:#d07711;">"Hello, world"</span><span>)
</span></code></pre>
<p>defined in that file, you can execute it from the command line via</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ fab hello
</span><span>hello
</span><span>
</span><span>Done.
</span></code></pre>
<p>It’s a great way to make Python scripts easily accessible from the command line,
but not very useful on its own. What makes Fabric shine are the <code>local</code> and
<code>run</code> functions. You can think of these functions as local and remote wrappers
around the <code>subprocess</code> module:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">from </span><span>fabric.api </span><span style="color:#72ab00;">import </span><span>local, run
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">hello</span><span>():
</span><span> cmd </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">'echo $HOSTNAME'
</span><span> </span><span style="color:#5597d6;">local</span><span>(cmd, </span><span style="color:#5597d6;">shell</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">True</span><span>)
</span><span> </span><span style="color:#5597d6;">run</span><span>(cmd, </span><span style="color:#5597d6;">shell</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">True</span><span>)
</span></code></pre>
<p>will print this on my local machine:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ fab hello
</span><span>[localhost] local: echo $HOSTNAME
</span><span>titan
</span><span>No hosts found. Please specify (single) host string for connection: bloerg.net
</span><span>[bloerg.net] run: echo $HOSTNAME
</span><span>[bloerg.net] out: openSUSE-121-64-minimal
</span><span>[bloerg.net] out:
</span><span>
</span><span>
</span><span>Done.
</span><span>Disconnecting from bloerg.net... done.
</span></code></pre>
<p>Because, I haven’t specified a host for the <code>run</code> command, Fabric will ask me
for a connection string. When a connection is established, the <code>echo</code> command is
executed on the bloerg.net host.</p>
<p>Now, that we can run shell code remotely, it’s pretty straightforward to fetch,
build and install code. Let’s assume I have to build and install ROOT and VTK on
a couple of machines from source. The following <code>fabfile.py</code> will take care of
this in an easy-to-understand and composable way (security considerations set
aside):</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">import </span><span>os
</span><span style="color:#72ab00;">from </span><span>fabric.api </span><span style="color:#72ab00;">import </span><span>run, env, cd
</span><span>
</span><span style="color:#5597d6;">VTK_SOURCE </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">'http://www.vtk.org/files/release/5.10/vtk-5.10.1.tar.gz'
</span><span style="color:#5597d6;">ROOT_SOURCE </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">'ftp://root.cern.ch/root/root_v5.34.05.source.tar.gz'
</span><span style="color:#5597d6;">ROOT_CFLAGS </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">' '</span><span>.</span><span style="color:#5597d6;">join</span><span>([</span><span style="color:#d07711;">'--enable-mathmore'</span><span>,
</span><span> </span><span style="color:#d07711;">'--enable-minuit2'</span><span>,
</span><span> </span><span style="color:#d07711;">'--enable-fftw3'</span><span>,
</span><span> </span><span style="color:#d07711;">'--enable-xml'</span><span>,
</span><span> </span><span style="color:#d07711;">'--prefix=/usr/local'</span><span>])
</span><span style="color:#5597d6;">BUILD_PATH </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">'/tmp'
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">wget</span><span>(</span><span style="color:#5597d6;">url</span><span>):
</span><span> </span><span style="color:#5597d6;">run</span><span>(</span><span style="color:#d07711;">'wget </span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">' </span><span style="color:#72ab00;">% </span><span>url)
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">untar</span><span>(</span><span style="color:#5597d6;">filename</span><span>):
</span><span> </span><span style="color:#5597d6;">run</span><span>(</span><span style="color:#d07711;">'tar xfz </span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">' </span><span style="color:#72ab00;">% </span><span>filename)
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">base</span><span>(</span><span style="color:#5597d6;">path</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span>os.path.</span><span style="color:#5597d6;">join</span><span>(</span><span style="color:#5597d6;">BUILD_PATH</span><span>, path)
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">fetch</span><span>():
</span><span> </span><span style="color:#72ab00;">with </span><span style="color:#5597d6;">cd</span><span>(</span><span style="color:#5597d6;">_base</span><span>(</span><span style="color:#d07711;">'.'</span><span>)):
</span><span> </span><span style="color:#5597d6;">wget</span><span>(</span><span style="color:#5597d6;">ROOT_SOURCE</span><span>)
</span><span> </span><span style="color:#5597d6;">untar</span><span>(os.path.</span><span style="color:#5597d6;">basename</span><span>(</span><span style="color:#5597d6;">ROOT_SOURCE</span><span>))
</span><span>
</span><span> </span><span style="color:#5597d6;">wget</span><span>(</span><span style="color:#5597d6;">VTK_SOURCE</span><span>)
</span><span> </span><span style="color:#5597d6;">untar</span><span>(os.path.</span><span style="color:#5597d6;">basename</span><span>(</span><span style="color:#5597d6;">VTK_SOURCE</span><span>))
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">install</span><span>():
</span><span> </span><span style="color:#5597d6;">sudo</span><span>(</span><span style="color:#d07711;">'zypper in gsl gsl-devel'</span><span>)
</span><span> </span><span style="color:#5597d6;">fetch</span><span>()
</span><span>
</span><span> n_cores </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">run</span><span>(</span><span style="color:#d07711;">'grep -c ^processor /proc/cpuinfo'</span><span>)
</span><span> make_cmd </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">'make -j</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">' </span><span style="color:#72ab00;">% </span><span>n_cores
</span><span>
</span><span> </span><span style="color:#72ab00;">with </span><span style="color:#5597d6;">cd</span><span>(</span><span style="color:#5597d6;">_base</span><span>(</span><span style="color:#d07711;">'root'</span><span>)):
</span><span> </span><span style="color:#5597d6;">run</span><span>(</span><span style="color:#d07711;">'./configure </span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">' </span><span style="color:#72ab00;">% </span><span style="color:#5597d6;">ROOT_CFLAGS</span><span>)
</span><span> </span><span style="color:#5597d6;">run</span><span>(make_cmd)
</span><span> </span><span style="color:#5597d6;">sudo</span><span>(</span><span style="color:#d07711;">'make install'</span><span>)
</span><span>
</span><span> </span><span style="color:#72ab00;">with </span><span style="color:#5597d6;">cd</span><span>(</span><span style="color:#5597d6;">_base</span><span>(</span><span style="color:#d07711;">'VTK5.10.1'</span><span>)):
</span><span> </span><span style="color:#5597d6;">run</span><span>(</span><span style="color:#d07711;">'cmake . -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release'</span><span>)
</span><span> </span><span style="color:#5597d6;">run</span><span>(make_cmd)
</span><span> </span><span style="color:#5597d6;">sudo</span><span>(</span><span style="color:#d07711;">'make install'</span><span>)
</span><span>
</span><span> </span><span style="color:#5597d6;">sudo</span><span>(</span><span style="color:#d07711;">'ldconfig'</span><span>)
</span></code></pre>
<p>To change into a certain directory for the time being, we <em>have to</em> use the <code>cd</code>
context manager because each command will be executed in its own SSH session.
As you can imagine, <code>sudo</code> will execute a command just like <code>run</code> but with super
user rights.</p>
<p>Defining and executing tasks is dead simple and the excellent <a href="http://docs.fabfile.org/en/1.6/#documentation">documentation</a>
helps mastering more advanced topics such as roles and parallel execution.</p>
<h2 id="alternatives">Alternatives</h2>
<p>Now, you could implement the same procedure in a shell script, copy that to each
machine and execute there. But here are some reasons why this is not the best
approach:</p>
<ol>
<li>Compared to shell languages, Python has a relatively sane syntax and a huge
standard library.</li>
<li>This approach doesn’t scale beyond a couple of machines. With Fabric you have
a single file in a central location.</li>
<li>The strong declarative style of individual tasks documents your intents.</li>
</ol>
<p>However, Fabric is not the end of the story. For massive infrastructure
automatization <a href="http://www.opscode.com/">Chef</a> and <a href="https://puppetlabs.com/">Puppet</a> are probably much better suited.</p>
Good code vs. usability2013-04-16T00:00:00+00:002013-04-16T00:00:00+00:00
Unknown
https://bloerg.net/posts/good-code-vs-usability/<p>Writing Python in a “Pythonic” way means adhering to conventions<sup class="footnote-reference"><a href="#1">1</a></sup> and using
the language in the most general way possible<sup class="footnote-reference"><a href="#2">2</a></sup>. I would even go so far and
say, that only Pythonic code is correct code. Unfortunately, this assumption
crumbles away when using Python code to interact with an end-user.</p>
<p>It is not uncommon to use Python itself as a domain specific language because it
is easy to learn and comes with a high quality, standard library and a vast
amount of third-party libraries that fill in even the smallest niche. This was
also the route that a colleague and I took for a piece of software that we are
currently developing. With this software, the user is expected to control
hardware devices from within an IPython shell.</p>
<p>Here is the dilemma: On the one hand, I try to write as Pythonic as possible, on
the other hand, I want to make it as easy as possible for the user to accomplish
a given task. For example, in a proper way, a function is called by appending
parentheses at the end of the name. But then, is the user aware of being inside
a programming environment? Instead of writing <code>device.show_info()</code>, the user
wants to type</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>>>> device
</span></code></pre>
<p>and investigate a device’s state. So, I abuse <code>__repr__</code> and return a string
that does not “look like a valid Python expression”, hence violating a very
basic requirements of the object “protocol”. Not very cool in my book.</p>
<p>I also added a function that shows information for all devices currently
defined. This is impossible to achieve without inspecting the frame stack,
iterating over all <code>globals()</code> and checking the type via <code>isinstance()</code>. This is
pure magic, but again very helpful for the user.</p>
<p>Without going on with more examples, the question quickly becomes: Where to draw
the line? How much should I mess with Python internals for the sake of a nice
user interface? Is even using Python as a DSL the right approach?</p>
<p>Maybe I am not experienced enough, but unfortunately I cannot answer these
questions.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>PEP8 is the prime example to enforce a consistent style through formal conventions whereas PEP20 is an informal guide line for writing good Python code.
<sup class="footnote-reference"><a href="#2">2</a></sup>: Often, iterating over iterables instead of manually constructing for loops is given as an introductory example for Pythonic code.</p>
</div>
Chaining Python futures2013-04-05T00:00:00+00:002013-04-05T00:00:00+00:00
Unknown
https://bloerg.net/posts/chaining-python-futures/<p><em>Update: I fixed a problem with the decorator, that leads to problems when
calling a decorated function several times or in a nested way. I also
shortened it a bit now.</em></p>
<p>One of Python 3.2’s new modules is <a href="http://docs.python.org/dev/library/concurrent.futures.html">concurrent.futures</a>. With it’s <em>executor</em>
and <em>future</em> abstractions, asynchronous, multi-threaded programming becomes less
of a headache:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">import </span><span>time
</span><span style="color:#72ab00;">from </span><span>concurrent.futures </span><span style="color:#72ab00;">import </span><span>ThreadPoolExecutor
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">read_and_block_for_a_while</span><span>():
</span><span> </span><span style="color:#72ab00;">pass
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">sleeps_a_bit</span><span>(</span><span style="color:#5597d6;">sleep_time</span><span>):
</span><span> time.</span><span style="color:#5597d6;">sleep</span><span>(sleep_time)
</span><span>
</span><span style="color:#72ab00;">with </span><span style="color:#5597d6;">ThreadPoolExecutor</span><span>(</span><span style="color:#5597d6;">max_workers</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">4</span><span>) </span><span style="color:#72ab00;">as </span><span>executor:
</span><span> future1 </span><span style="color:#72ab00;">= </span><span>executor.</span><span style="color:#5597d6;">submit</span><span>(read_and_block_for_a_while)
</span><span> future2 </span><span style="color:#72ab00;">= </span><span>executor.</span><span style="color:#5597d6;">submit</span><span>(sleeps_a_bit, </span><span style="color:#b3933a;">1.5</span><span>)
</span><span>
</span><span> </span><span style="color:#7f8989;"># Now, we can continue doing other stuff ...
</span><span>
</span><span> </span><span style="color:#7f8989;"># But at some point we need the result and we will block until the result is
</span><span> </span><span style="color:#7f8989;"># delivered.
</span><span> data </span><span style="color:#72ab00;">= </span><span>future1.</span><span style="color:#5597d6;">result</span><span>()
</span></code></pre>
<p>As you can see, a future combines a thread of execution, the threads current
state and methods to query and wait for results. You can pass it around as if it
were a value and you can add callbacks to be notified when a result is ready –
without ever missing the result as in message-based systems.</p>
<p>Now, passing futures around sounds like fun but becomes quite awkward to use.
This is because functions have to expect future arguments and call the
<code>result()</code> method when they need an argument’s value. But most of the time we
want to write “normal” functions in which we don’t want to take care about
futures. Unfortunately, the futures API is modeled a little bit too close after
Java’s <a href="http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Future.html">futures</a>, which is why we have to get our hands a little bit dirty
with <a href="https://gist.github.com/matze/5325713">this promise decorator</a>:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">from </span><span>concurrent.futures </span><span style="color:#72ab00;">import </span><span>Future, ThreadPoolExecutor
</span><span>
</span><span>
</span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">promise</span><span>(</span><span style="font-style:italic;color:#b06936;">object</span><span>):
</span><span>
</span><span> executor </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">ThreadPoolExecutor</span><span>(</span><span style="color:#5597d6;">max_workers</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">100</span><span>)
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#b39f04;">__init__</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#5597d6;">func</span><span>):
</span><span> </span><span style="color:#5597d6;">self</span><span>.func </span><span style="color:#72ab00;">= </span><span>func
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">resolve</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">args</span><span>, </span><span style="color:#72ab00;">**</span><span style="color:#5597d6;">kwargs</span><span>):
</span><span> resolved_args </span><span style="color:#72ab00;">= </span><span>[]
</span><span> resolved_kwargs </span><span style="color:#72ab00;">= </span><span>{}
</span><span>
</span><span> </span><span style="color:#72ab00;">for </span><span>i, arg </span><span style="color:#72ab00;">in </span><span style="color:#b39f04;">enumerate</span><span>(args):
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#b39f04;">isinstance</span><span>(arg, Future):
</span><span> resolved_args.</span><span style="color:#5597d6;">append</span><span>(arg.</span><span style="color:#5597d6;">result</span><span>())
</span><span> </span><span style="color:#72ab00;">else</span><span>:
</span><span> resolved_args.</span><span style="color:#5597d6;">append</span><span>(arg)
</span><span>
</span><span> </span><span style="color:#72ab00;">for </span><span>kw, arg </span><span style="color:#72ab00;">in </span><span>kwargs.</span><span style="color:#5597d6;">items</span><span>():
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#b39f04;">isinstance</span><span>(arg, Future):
</span><span> resolved_kwargs[kw] </span><span style="color:#72ab00;">= </span><span>arg.</span><span style="color:#5597d6;">result</span><span>()
</span><span> </span><span style="color:#72ab00;">else</span><span>:
</span><span> resolved_kwargs[kw] </span><span style="color:#72ab00;">= </span><span>arg
</span><span>
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">func</span><span>(</span><span style="color:#72ab00;">*</span><span>resolved_args, </span><span style="color:#72ab00;">**</span><span>resolved_kwargs)
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#b39f04;">__call__</span><span>(</span><span style="color:#5597d6;">self</span><span>, </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">args</span><span>, </span><span style="color:#72ab00;">**</span><span style="color:#5597d6;">kwargs</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#5597d6;">self</span><span>.executor.</span><span style="color:#5597d6;">submit</span><span>(</span><span style="color:#5597d6;">self</span><span>.resolve, </span><span style="color:#72ab00;">*</span><span>args, </span><span style="color:#72ab00;">**</span><span>kwargs)
</span></code></pre>
<p>The decorator creates a new future that calls <code>resolve()</code>. This method will call
<code>.result()</code> on all regular and keyword arguments that are of future type <sup class="footnote-reference"><a href="#1">1</a></sup>.
When done, the wrapped function will be called with actual values.</p>
<p>Because a C++’s <code>std::promise</code> works in a similar way, I also called the
decorator <code>promise</code>. The following example shows how it is used for chaining:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span>@</span><span style="color:#5597d6;">promise
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">initial</span><span>(</span><span style="color:#5597d6;">x</span><span>):
</span><span> time.</span><span style="color:#5597d6;">sleep</span><span>(</span><span style="color:#b3933a;">1.0</span><span>)
</span><span> </span><span style="color:#72ab00;">return </span><span>x
</span><span>
</span><span>@</span><span style="color:#5597d6;">promise
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">add</span><span>(</span><span style="color:#5597d6;">x</span><span>, </span><span style="color:#5597d6;">y</span><span>):
</span><span> </span><span style="color:#72ab00;">return </span><span>x </span><span style="color:#72ab00;">+ </span><span>y
</span><span>
</span><span>future </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">add</span><span>(</span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#5597d6;">initial</span><span>(</span><span style="color:#b3933a;">2</span><span>))
</span><span style="color:#7f8989;"># do something else ...
</span><span style="color:#b39f04;">print</span><span>(future.</span><span style="color:#5597d6;">result</span><span>())
</span></code></pre>
<p>Unfortunately, there are three caveats with this approach. First, the decorator
class keeps a reference to the executor. This makes it a bit uncomfortable when
using the <code>with</code> statement:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">with </span><span style="color:#5597d6;">ThreadPoolExecutor</span><span>(</span><span style="color:#5597d6;">max_workers</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">2</span><span>) </span><span style="color:#72ab00;">as </span><span>e:
</span><span> promise.executor </span><span style="color:#72ab00;">= </span><span>e
</span><span> </span><span style="color:#b3933a;">...
</span></code></pre>
<p>Second, the promise decorator only works with a <code>ThreadPoolExecutor</code>. As stated
in the documentation:</p>
<blockquote>
<p>Calling Executor or Future methods from a callable submitted to a
ProcessPoolExecutor will result in deadlock.</p>
</blockquote>
<p>… which is exactly what we do. This is a little bit unfortunate, because the
<code>Executor</code> interface does not restrict this kind of usage and I would expect all
implementations of an interface to behave the same.</p>
<p>Last but not least, we are serializing the calls by waiting for all input
futures to be finished. That means, you won’t gain much from deep calls because
they don’t run in parallel. On the other hand, you will see no difference in
run-time if you compute</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b39f04;">print</span><span>(</span><span style="color:#5597d6;">initial</span><span>(</span><span style="color:#b3933a;">1</span><span>).result)
</span></code></pre>
<p>or</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#b39f04;">print</span><span>(</span><span style="color:#5597d6;">add</span><span>(</span><span style="color:#5597d6;">add</span><span>(</span><span style="color:#5597d6;">initial</span><span>(</span><span style="color:#b3933a;">1</span><span>), </span><span style="color:#5597d6;">initial</span><span>(</span><span style="color:#b3933a;">2</span><span>)), </span><span style="color:#5597d6;">initial</span><span>(</span><span style="color:#b3933a;">3</span><span>)).</span><span style="color:#5597d6;">result</span><span>())
</span></code></pre>
<p>Both will take exactly one second.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>I know that this is <a href="http://www.canonical.org/~kragen/isinstance/">not</a> Pythonic, but there is no other way to ensure that we are calling a future’s <code>result()</code> method.</p>
</div>
PyCon 2013 talks2013-03-24T00:00:00+00:002013-03-24T00:00:00+00:00
Unknown
https://bloerg.net/posts/pycon-2013-talks/<p>Apart from <em><a href="http://amandablumwords.wordpress.com/2013/03/21/3/">that</a></em> incident, <a href="https://us.pycon.org/2013/">PyCon US 2013</a> featured a vast amount of high
quality <a href="http://pyvideo.org/category/33/pycon-us-2013">talks</a>. From the <del>84</del> <del>97</del> <del>101</del>
<del>110</del> 117 videos, I can highly recommend the talks concerning
<a href="https://bloerg.net/posts/pycon-2013-talks/#testing">testing</a>, <a href="https://bloerg.net/posts/pycon-2013-talks/#documentation">documenting</a> and <a href="https://bloerg.net/posts/pycon-2013-talks/#design">design</a>.
Unfortunately, neither the conference site nor Pyvideo provide a classification
of the talks, so here’s one from me:</p>
<h2 id="testing">Testing</h2>
<ul>
<li><a href="http://pyvideo.org/video/1674/getting-started-with-automated-testing">Getting started with automated testing</a></li>
<li><a href="http://pyvideo.org/video/1690/things-to-make-writing-tests-easier">Things to make writing tests easier</a> ♥</li>
<li><a href="http://pyvideo.org/video/1699/mobile-application-testing-with-python-and-seleni">Mobile Application Testing with Python and Selenium</a></li>
</ul>
<h2 id="documentation">Documentation</h2>
<ul>
<li><a href="http://pyvideo.org/video/1703/better-documentation-through-automation-creating">Better Documentation Through Automation: Creating Sphinx Extensions</a> ♥</li>
<li><a href="http://pyvideo.org/video/1795/write-the-docs">Write the Docs</a> ♥</li>
<li><a href="http://pyvideo.org/video/1786/if-you-code-you-should-write">If You Code, You Should Write</a> ♥</li>
</ul>
<h2 id="design">Design</h2>
<ul>
<li><a href="http://pyvideo.org/video/1760/encapsulation-with-descriptors">Encapsulation with descriptors</a> ♥</li>
<li><a href="http://pyvideo.org/video/1789/dynamic-code-patterns-extending-your-application">Dynamic Code Patterns: Extending Your Applications with Plugins</a></li>
<li><a href="http://pyvideo.org/video/1705/api-design-for-library-authors">API Design for Library Authors</a> ♥</li>
<li><a href="http://pyvideo.org/video/1799/functional-programming-with-python">Functional Programming with Python</a></li>
<li><a href="http://pyvideo.org/video/1681/so-easy-you-can-even-do-it-in-javascript-event-d">So Easy You Can Even Do It in JavaScript: Event-Driven Architecture for Regular Programmers</a></li>
<li><a href="http://pyvideo.org/video/1680/how-to-except-when-youre-excepting">How to Except When You’re Excepting</a></li>
<li><a href="http://pyvideo.org/video/1738/good-enough-is-good-enough">“Good enough” is good enough!</a></li>
<li><a href="http://pyvideo.org/video/1762/using-futures-for-async-gui-programming-in-python">Using futures for async GUI programming in Python 3.3</a> ♥</li>
<li><a href="http://pyvideo.org/video/1684/the-end-of-object-inheritance-the-beginning-of">The End Of Object Inheritance & The Beginning Of A New Modularity</a> ♥</li>
<li><a href="http://pyvideo.org/video/1746/internationalization-and-localization-done-right">Internationalization and Localization Done Right</a></li>
<li><a href="http://pyvideo.org/video/1780/transforming-code-into-beautiful-idiomatic-pytho">Transforming Code into Beautiful, Idiomatic Python</a> ♥</li>
<li><a href="http://pyvideo.org/video/1772/let-them-configure">Let Them Configure!</a></li>
<li><a href="http://pyvideo.org/video/1676/the-naming-of-ducks-where-dynamic-types-meet-sma">The Naming of Ducks: Where Dynamic Types Meet Smart Conventions</a></li>
<li><a href="http://pyvideo.org/video/1785/python-for-humans-1">Python for Humans</a> ♥</li>
<li><a href="http://pyvideo.org/video/1678/composability-through-multiple-inheritance">Composability Through Multiple Inheritance</a></li>
</ul>
<h2 id="behind-the-scenes">Behind the scenes</h2>
<ul>
<li><a href="http://pyvideo.org/video/1850/a-hands-on-introduction-to-python-for-beginning-p">A hands-on introduction to Python for beginning programmers</a></li>
<li><a href="http://pyvideo.org/video/1707/how-import-works">How Import Works</a></li>
<li><a href="http://pyvideo.org/video/1777/all-singing-all-dancing-python-bytecode">All-Singing All-Dancing Python Bytecode</a></li>
<li><a href="http://pyvideo.org/video/1768/the-guts-of-unicode-in-python">The Guts of Unicode in Python</a></li>
<li><a href="http://pyvideo.org/video/1730/python-33-trust-me-its-better-than-27">Python 3.3: Trust Me, It’s Better than 2.7</a></li>
<li><a href="http://pyvideo.org/video/1729/python-a-toy-language">Python: A “Toy” Language</a></li>
<li><a href="http://pyvideo.org/video/1704/why-you-should-use-python-3-for-text-processing">Why you should use Python 3 for text processing</a></li>
<li><a href="http://pyvideo.org/video/1758/loop-like-a-native-while-for-iterators-genera">Loop like a native: while, for, iterators, generators</a></li>
<li><a href="http://pyvideo.org/video/1716/python-3-metaprogramming">Python 3 Metaprogramming</a></li>
<li><a href="http://pyvideo.org/video/1694/so-you-want-to-write-an-interpreter">So you want to write an interpreter?</a></li>
<li><a href="http://pyvideo.org/video/1779/pythons-class-development-toolkit">Python’s Class Development Toolkit</a></li>
<li><a href="http://pyvideo.org/video/1659/python-epiphanies-0">Python Epiphanies</a></li>
</ul>
<h2 id="community-and-people">Community and People</h2>
<ul>
<li><a href="http://pyvideo.org/video/1742/how-not-to-build-an-oss-community">How (Not) To Build An OSS Community</a></li>
<li><a href="http://pyvideo.org/video/1687/scaling-community-diversity-outreach">Scaling community diversity outreach</a></li>
<li><a href="http://pyvideo.org/video/1791/what-is-the-python-software-foundation">What is the Python Software Foundation?</a></li>
<li><a href="http://pyvideo.org/video/1782/sink-or-swim-5-life-jackets-to-throw-to-new-code">Sink or swim: 5 life jackets to throw to New Coders</a></li>
<li><a href="http://pyvideo.org/video/1754/who-are-we-a-sociological-analysis-of-the-indige">Who are we? A sociological analysis of the indigenous Python tribe</a></li>
<li><a href="http://pyvideo.org/video/1732/designers-developers-collaborating-on-your-pyt">Designers + Developers: Collaborating on your Python project</a></li>
</ul>
<h3 id="teaching">Teaching</h3>
<ul>
<li><a href="http://pyvideo.org/video/1697/what-teachers-really-need-from-us">What teachers really need from us</a></li>
<li><a href="http://pyvideo.org/video/1744/teaching-with-the-ipython-notebook">Teaching with the IPython Notebook</a></li>
<li><a href="http://pyvideo.org/video/1775/community-event-planning">Community Event Planning</a></li>
<li><a href="http://pyvideo.org/video/1783/edupsych-theory-for-python-hackers-a-whirlwind-o">EduPsych Theory for Python Hackers: A Whirlwind Overview</a></li>
<li><a href="http://pyvideo.org/video/1793/lessons-learned-in-teaching-python">Lessons Learned in Teaching Python</a></li>
</ul>
<h2 id="tutorials">Tutorials</h2>
<ul>
<li><a href="http://pyvideo.org/video/1725/learn-python-through-public-data-hacking">Learn Python Through Public Data Hacking</a></li>
<li><a href="http://pyvideo.org/video/1651/python-for-programmers-a-project-based-tutorial">Python for Programmers: A Project-Based Tutorial</a></li>
</ul>
<h2 id="optimization">Optimization</h2>
<ul>
<li><a href="http://pyvideo.org/video/1739/pypy-without-the-gil">PyPy without the GIL</a></li>
<li><a href="http://pyvideo.org/video/1692/cython-vs-swig-fight">Cython vs SWIG, Fight!</a></li>
<li><a href="http://pyvideo.org/video/1770/python-profiling">Python Profiling</a></li>
<li><a href="http://pyvideo.org/video/1765/blame-it-on-caesar-what-you-need-to-know-about-d">Blame it on Caesar: What you need to know about dates, times and time zones</a></li>
<li><a href="http://pyvideo.org/video/1781/awesome-big-data-algorithms">Awesome Big Data Algorithms</a></li>
</ul>
<h3 id="parallel-computing">Parallel computing</h3>
<ul>
<li><a href="http://pyvideo.org/video/1720/applied-parallel-computing-with-python">Applied Parallel Computing with Python</a></li>
<li><a href="http://pyvideo.org/video/1652/ipython-in-depth-high-productivity-interactive-a-0">IPython in-depth: high-productivity interactive and parallel python</a></li>
</ul>
<h2 id="web">Web</h2>
<ul>
<li><a href="http://pyvideo.org/video/1677/how-the-internet-works">How the Internet works</a></li>
<li><a href="http://pyvideo.org/video/1663/going-from-web-pages-to-web-apps-with-python">Going from web pages to web apps with Python</a></li>
<li><a href="http://pyvideo.org/video/1696/dispatch-magic">Dispatch Magic</a></li>
<li><a href="http://pyvideo.org/video/1773/making-apache-suck-less-for-hosting-python-web-ap">Making Apache suck less for hosting Python web applications.</a></li>
<li><a href="http://pyvideo.org/video/1798/make-more-responsive-web-applications-with-socket">Make More Responsive Web Applications with SocketIO and gevent</a></li>
<li><a href="http://pyvideo.org/video/1740/twisted-logic-endpoints-and-why-you-shouldnt-be">Twisted Logic: Endpoints and Why You Shouldn’t Be Scared of Twisted</a></li>
<li><a href="http://pyvideo.org/video/1792/asset-management-in-python">Asset Management in Python</a></li>
<li><a href="http://pyvideo.org/video/1685/scrapy-it-gets-the-web">Scrapy: it GETs the web</a></li>
<li><a href="http://pyvideo.org/video/1723/rapid-web-prototyping-with-lightweight-tools">Rapid Web Prototyping with Lightweight Tools</a></li>
</ul>
<h3 id="django">Django</h3>
<ul>
<li><a href="http://pyvideo.org/video/1717/effective-django-0">Effective Django</a></li>
<li><a href="http://pyvideo.org/video/1787/porting-django-apps-to-python-3">Porting Django apps to Python 3</a></li>
<li><a href="http://pyvideo.org/video/1752/location-location-location">Location, Location, Location</a></li>
<li><a href="http://pyvideo.org/video/1755/customizing-the-django-admin-the-how-and-the-why">Customizing the Django Admin: The How and the Why</a></li>
<li><a href="http://pyvideo.org/video/1657/fully-test-driven-web-development-with-django-and">Fully Test-Driven Web Development with Django and Selenium</a></li>
</ul>
<h3 id="deployment">Deployment</h3>
<ul>
<li><a href="http://pyvideo.org/video/1776/deploy-your-python-app-in-5-min-with-a-paas">Deploy your Python app in 5 min with a PaaS</a></li>
<li><a href="http://pyvideo.org/video/1727/solid-python-application-deployments-for-everybod">Solid Python Application Deployments For Everybody</a></li>
</ul>
<h2 id="testimonials">Testimonials</h2>
<ul>
<li><a href="http://pyvideo.org/video/1743/python-at-netflix">Python at Netflix</a></li>
<li><a href="http://pyvideo.org/video/1671/messaging-at-scale-at-instagram">Messaging at Scale at Instagram</a></li>
<li><a href="http://pyvideo.org/video/1682/this-old-video-site-how-pbs-streams-video-and">This Old Video Site: How PBS streams video - and you can too!</a></li>
<li><a href="http://pyvideo.org/video/1700/making-disqus-realtime-0">Making DISQUS Realtime.</a></li>
<li><a href="http://pyvideo.org/video/1759/visualizing-github-part-i-data-to-information">Visualizing Github, Part I: Data to Information</a></li>
<li><a href="http://pyvideo.org/video/1766/visualizing-github-part-ii-information-to-meani">Visualizing Github, Part II: Information to Meaning</a></li>
</ul>
<h2 id="security">Security</h2>
<ul>
<li><a href="http://pyvideo.org/video/1764/beyond-passwords-secure-authentication-with-mozi-0">Beyond Passwords: Secure Authentication with Mozilla Persona</a></li>
<li><a href="http://pyvideo.org/video/1778/crypto-101">Crypto 101</a></li>
<li><a href="http://pyvideo.org/video/1698/death-by-a-thousand-leaks-what-statically-analys">Death by a thousand leaks: what statically-analysing 370 Python extensions looks like</a></li>
<li><a href="http://pyvideo.org/video/1691/pyramid-auth-is-hard-lets-ride-bikes">Pyramid Auth Is Hard, Let’s Ride Bikes</a></li>
</ul>
<h2 id="applications">Applications</h2>
<ul>
<li><a href="http://pyvideo.org/video/1706/plover-thought-to-text-at-240-wpm">Plover: Thought to Text at 240 WPM</a></li>
<li><a href="http://pyvideo.org/video/1736/worry-free-parsers-with-parsley">Worry-Free Parsers with Parsley</a></li>
<li><a href="http://pyvideo.org/video/1702/building-an-image-processing-pipeline-with-python">Building an image processing pipeline with Python</a></li>
<li><a href="http://pyvideo.org/video/1751/building-full-stack-scientific-applications-in-py">Building full-stack scientific applications in Python</a></li>
<li><a href="http://pyvideo.org/video/1718/introduction-to-pygame">Introduction to PyGame</a></li>
<li><a href="http://pyvideo.org/video/1711/introduction-to-sqlalchemy">Introduction to SQLAlchemy</a></li>
<li><a href="http://pyvideo.org/video/1767/the-sqlalchemy-session-in-depth-0">The SQLAlchemy Session - In Depth</a></li>
<li><a href="http://pyvideo.org/video/1701/kivy-building-gui-and-mobile-apps-with-python">Kivy: Building GUI and Mobile apps with Python</a></li>
<li><a href="http://pyvideo.org/video/1719/advanced-machine-learning-with-scikit-learn">Advanced Machine Learning with scikit-learn</a></li>
<li><a href="http://pyvideo.org/video/1714/analyzing-social-networks-with-python">Analyzing Social Networks with Python</a></li>
<li><a href="http://pyvideo.org/video/1735/using-python-to-code-by-voice">Using Python to Code by Voice</a></li>
<li><a href="http://pyvideo.org/video/1790/mto-on-blast-using-pythons-natural-language-too">MTO On Blast: Using Python’s Natural Language Toolkit to Model Gossip Blogs</a></li>
<li><a href="http://pyvideo.org/video/1771/realtime-tracking-and-mapping-of-geographic-objec">Realtime Tracking and Mapping of Geographic Objects using Python</a></li>
<li><a href="http://pyvideo.org/video/1688/whos-there-home-automation-with-arduinorasp">Who’s there? - Home Automation with Arduino/RaspberryPi</a></li>
<li><a href="http://pyvideo.org/video/1769/music-theory-and-performance-analysis-with-sebast">Music Theory and Performance Analysis with Sebastian and Czerny</a></li>
<li><a href="http://pyvideo.org/video/1796/simplecv-computer-vision-using-python">SimpleCV - Computer Vision using Python</a></li>
<li><a href="http://pyvideo.org/video/1784/elasticsearch-part-1-indexing-and-querying">Elasticsearch (Part 1): Indexing and Querying</a></li>
<li><a href="http://pyvideo.org/video/1761/fighting-cancer-with-python">Fighting Cancer with Python</a></li>
<li><a href="http://pyvideo.org/video/1788/whoosh-the-open-source-python-search-library">Whoosh, the open-source Python search library</a></li>
<li><a href="http://pyvideo.org/video/1748/use-curses-dont-swear">Use curses, don’t swear</a></li>
<li><a href="http://pyvideo.org/video/1655/an-introduction-to-scikit-learn-machine-learning">An Introduction to scikit-learn: Machine Learning in Python</a></li>
<li><a href="http://pyvideo.org/video/1724/bayesian-statistics-made-simple">Bayesian statistics made simple</a></li>
<li><a href="http://pyvideo.org/video/1741/measuring-and-modeling-the-complexity-of-children">Measuring and modeling the complexity of children’s books</a></li>
</ul>
<h3 id="data-analysis">Data analysis</h3>
<ul>
<li><a href="http://pyvideo.org/video/1654/a-beginners-introduction-to-pydata-how-to-build">A beginner’s introduction to Pydata: how to build a minimal recommendation engine.</a></li>
<li><a href="http://pyvideo.org/video/1656/python-for-data-analysis">Python for Data Analysis</a></li>
<li><a href="http://pyvideo.org/video/1745/server-log-analysis-with-pandas-0">Server Log Analysis with Pandas</a></li>
</ul>
<h2 id="meta">Meta</h2>
<ul>
<li><a href="http://pyvideo.org/video/1848/opening-statements">Opening Statements</a></li>
<li><a href="http://pyvideo.org/video/1666/keynote-0">Keynote Van Lindberg</a></li>
<li><a href="http://pyvideo.org/video/1667/keynote-1">Keynote Guido van Rossum</a></li>
<li><a href="http://pyvideo.org/video/1668/keynote-2">Keynote Eben Upton</a></li>
<li><a href="http://pyvideo.org/video/1669/keynote-3">Keynote Raymond Hettinger</a></li>
<li><a href="http://pyvideo.org/video/1763/closing-address">Closing Address</a></li>
<li><a href="http://pyvideo.org/video/1731/panel-directions-for-packaging">Panel: Directions for Packaging</a></li>
<li><a href="http://pyvideo.org/video/1675/pythons-law">Python’s Law</a></li>
<li><a href="http://pyvideo.org/video/1753/logical-failures">Logical Failures</a></li>
<li><a href="http://pyvideo.org/video/1852/the-future-of-linux-containers">Lightning Talk - The future of Linux Containers</a></li>
<li><a href="http://pyvideo.org/video/1853/friday-evening-lightning-talks">Friday evening lightning talks</a></li>
<li><a href="http://pyvideo.org/video/1854/saturday-morning-lightning-talks-0">Saturday morning lightning talks</a></li>
</ul>
Using Kramdown instead of Maruku2013-03-07T00:00:00+00:002013-03-07T00:00:00+00:00
Unknown
https://bloerg.net/posts/using-kramdown-instead-of-maruku/<p><em>Update January 23rd, 2015</em>: <a href="http://paulrobertlloyd.com/">Paul Robert Lloyd</a> improved the converter with
his <a href="https://github.com/paulrobertlloyd/jekyll-pypedown">Pypedown</a> plugin. It also sports the Typogruby extension for some visual
juice.</p>
<p><em>Update July 24th, 2014</em>: Matthijs van den Bos <a href="http://vandenbos.org/new-jekyll-plugin-kramdown-with-pygments/">improved</a> Juan’s plugin to work
with Jekyll 2.x.</p>
<p><em>Update May 7th, 2014</em>: As of Jekyll version 2.0.0, Kramdown is Jekyll’s default
Markdown converter. However, it’s still using CodeRay.</p>
<p><em>Update August 14th, 2013</em>: The code as shown below is not working with Jekyll versions
>= 1.0. See <a href="/2013/05/06/moving-to-jekyll-1-0.html">this post</a> how to update your Jekyll installation to version 1.0.
<a href="http://navarroj.com/research/">Juan Antonio Navarro Pérez</a> kindly put a fixed
version of the code on <a href="https://github.com/navarroj/krampygs">GitHub</a>.</p>
<hr />
<p>After I messed around with Maruku for some time now, I became fed up with its
limitations such as missing fenced code blocks. Without these, you have to use
the unwieldy <a href="https://github.com/mojombo/jekyll/wiki/Liquid-Extensions">Liquid tags</a> to denote a piece of code. Fortunuately, there are
many alternative Markdown converters. To decide which one to use in the future,
I came up with this (incomplete) feature matrix:</p>
<table><thead><tr><th>Feature</th><th style="text-align: center">Maruku</th><th style="text-align: center">Redcarpet</th><th style="text-align: center">Kramdown</th><th style="text-align: center">Pandoc</th></tr></thead><tbody>
<tr><td>Fenced code blocks</td><td style="text-align: center">–</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td></tr>
<tr><td>Footnotes</td><td style="text-align: center">✓</td><td style="text-align: center">–</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td></tr>
<tr><td>Jekyll integration</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td><td style="text-align: center">–</td></tr>
<tr><td>Header IDs</td><td style="text-align: center">✓</td><td style="text-align: center">–</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td></tr>
<tr><td>Inline math</td><td style="text-align: center">–</td><td style="text-align: center">–</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td></tr>
<tr><td>Abbreviations</td><td style="text-align: center">✓</td><td style="text-align: center">–</td><td style="text-align: center">✓</td><td style="text-align: center">✓</td></tr>
<tr><td>Pygments</td><td style="text-align: center">✓</td><td style="text-align: center">–</td><td style="text-align: center">–</td><td style="text-align: center">✓</td></tr>
</tbody></table>
<p>At first glance, Kramdown wins hands down (even ahead of my beloved Pandoc).
But there is one slight problem: When blogging with Jekyll, you have to use
<a href="http://coderay.rubychan.de/">CodeRay</a> for syntax highlighting or none at all. Because I favor <a href="http://pygments.org/">Pygments</a>
due to the broader range of supported languages, I had to update the Markdown
converter plugin that I presented in the <a href="/2013/03/03/typographic-improvements-for-jekyll.html">last post</a>.</p>
<p>For this, we have to subclass Kramdown’s HTML converter to use Pygments on code
blocks:</p>
<pre data-lang="ruby" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#72ab00;">require </span><span style="color:#d07711;">'kramdown'
</span><span style="color:#72ab00;">require </span><span style="color:#d07711;">'pygments'
</span><span style="color:#72ab00;">require </span><span style="color:#d07711;">'typogruby'
</span><span>
</span><span style="color:#72ab00;">module </span><span style="color:#c23f31;">Kramdown
</span><span> </span><span style="color:#72ab00;">module </span><span style="color:#c23f31;">Converter
</span><span> </span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">Pygs </span><span>< </span><span style="font-style:italic;color:#b06936;">Html
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">convert_codeblock</span><span>(</span><span style="color:#5597d6;">el</span><span>, </span><span style="color:#5597d6;">indent</span><span>)
</span><span> </span><span style="color:#72ab00;">attr =</span><span> el.</span><span style="color:#72ab00;">attr</span><span>.</span><span style="color:#b39f04;">dup
</span><span> lang </span><span style="color:#72ab00;">=</span><span> extract_code_language!(</span><span style="color:#72ab00;">attr</span><span>)
</span><span> </span><span style="color:#72ab00;">if</span><span> lang
</span><span> add_code_tags(
</span><span> </span><span style="color:#a2a001;">Pygments</span><span>.highlight(el.value,
</span><span> </span><span style="color:#b3933a;">:lexer </span><span>=> lang,
</span><span> </span><span style="color:#b3933a;">:options </span><span>=> { </span><span style="color:#b3933a;">:encoding </span><span>=> </span><span style="color:#d07711;">'utf-8' </span><span>})
</span><span> )
</span><span> </span><span style="color:#72ab00;">else
</span><span> </span><span style="color:#d07711;">"<pre><code></span><span>#{el.value}</span><span style="color:#d07711;"></code></pre>"
</span><span> </span><span style="color:#72ab00;">end
</span><span> </span><span style="color:#72ab00;">end
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">add_code_tags</span><span>(</span><span style="color:#5597d6;">code</span><span>)
</span><span> code </span><span style="color:#72ab00;">=</span><span> code.</span><span style="color:#b39f04;">sub</span><span>(</span><span style="color:#c49a39;">/<pre>/</span><span>,</span><span style="color:#d07711;">'<pre><code>'</span><span>)
</span><span> code </span><span style="color:#72ab00;">=</span><span> code.</span><span style="color:#b39f04;">sub</span><span>(</span><span style="color:#c49a39;">/<</span><span style="color:#108f3d;">\/</span><span style="color:#c49a39;">pre>/</span><span>,</span><span style="color:#d07711;">"</code></pre>"</span><span>)
</span><span> </span><span style="color:#72ab00;">end
</span><span> </span><span style="color:#72ab00;">end
</span><span> </span><span style="color:#72ab00;">end
</span><span style="color:#72ab00;">end
</span></code></pre>
<p>Now, we can replace the Maruku converter with our customized Kramdown converter:</p>
<pre data-lang="ruby" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#72ab00;">module </span><span style="color:#c23f31;">Jekyll
</span><span> </span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">MarkdownConverter </span><span>< </span><span style="color:#a2a001;">Jekyll</span><span style="font-style:italic;color:#b06936;">::Converter
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">convert</span><span>(</span><span style="color:#5597d6;">content</span><span>)
</span><span> setup
</span><span>
</span><span> html </span><span style="color:#72ab00;">= </span><span style="color:#a2a001;">Kramdown</span><span>::</span><span style="color:#a2a001;">Document</span><span>.</span><span style="color:#72ab00;">new</span><span>(content, {
</span><span> </span><span style="color:#b3933a;">:auto_ids </span><span>=> </span><span style="color:#5597d6;">@config</span><span>[</span><span style="color:#d07711;">'kramdown'</span><span>][</span><span style="color:#d07711;">'auto_ids'</span><span>],
</span><span> </span><span style="color:#b3933a;">:footnote_nr </span><span>=> </span><span style="color:#5597d6;">@config</span><span>[</span><span style="color:#d07711;">'kramdown'</span><span>][</span><span style="color:#d07711;">'footnote_nr'</span><span>],
</span><span> </span><span style="color:#b3933a;">:entity_output </span><span>=> </span><span style="color:#5597d6;">@config</span><span>[</span><span style="color:#d07711;">'kramdown'</span><span>][</span><span style="color:#d07711;">'entity_output'</span><span>],
</span><span> </span><span style="color:#b3933a;">:toc_levels </span><span>=> </span><span style="color:#5597d6;">@config</span><span>[</span><span style="color:#d07711;">'kramdown'</span><span>][</span><span style="color:#d07711;">'toc_levels'</span><span>],
</span><span> </span><span style="color:#b3933a;">:smart_quotes </span><span>=> </span><span style="color:#5597d6;">@config</span><span>[</span><span style="color:#d07711;">'kramdown'</span><span>][</span><span style="color:#d07711;">'smart_quotes'</span><span>]
</span><span> }).to_pygs
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#a2a001;">Typogruby</span><span>.improve(html)
</span><span> </span><span style="color:#72ab00;">end
</span><span> </span><span style="color:#72ab00;">end
</span><span style="color:#72ab00;">end
</span></code></pre>
<p>Note, that Kramdown uses weird dynamic name resolution (the method call
<code>to_pygs</code> in this case) to hook the subclassed <code>Pygs</code> class into the conversion
process. For my taste, this goes a bit too far but that’s the way it is.</p>
Typographic improvements for Jekyll2013-03-03T00:00:00+00:002013-03-03T00:00:00+00:00
Unknown
https://bloerg.net/posts/typographic-improvements-for-jekyll/<p>I <a href="/2012/11/23/getting-the-most-out-of-stock-latex.html#fonts">babbled</a> about improving the typography of LaTeX output with some simple
adjustments. Now with <a href="http://avdgaag.github.com/typogruby/">Typogruby</a>, we can enhance the HTML for a Jekyll blog
in a similar way.</p>
<h2 id="integrating-typogruby-into-jekyll">Integrating Typogruby into Jekyll</h2>
<p>Integrating Typogruby is as simple as subclassing Jekyll’s <code>Converter</code> and
calling Typogruby’s <code>improve</code> method in the <code>convert</code> method. Put this snippet</p>
<pre data-lang="ruby" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-ruby "><code class="language-ruby" data-lang="ruby"><span style="color:#72ab00;">require </span><span style="color:#d07711;">'typogruby'
</span><span>
</span><span style="color:#72ab00;">module </span><span style="color:#c23f31;">Jekyll
</span><span> </span><span style="color:#72ab00;">class </span><span style="color:#c23f31;">MarkdownConverter </span><span>< </span><span style="color:#a2a001;">Jekyll</span><span style="font-style:italic;color:#b06936;">::Converter
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">convert</span><span>(</span><span style="color:#5597d6;">content</span><span>)
</span><span> setup
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#a2a001;">Typogruby</span><span>.improve(</span><span style="color:#a2a001;">Maruku</span><span>.</span><span style="color:#72ab00;">new</span><span>(content).to_html)
</span><span> </span><span style="color:#72ab00;">end
</span><span> </span><span style="color:#72ab00;">end
</span><span style="color:#72ab00;">end
</span></code></pre>
<p>into your <code>_plugins</code> directory, and you are good to go.</p>
<h2 id="small-caps">Small caps</h2>
<p>Typogruby adds <code>span.caps</code> tags around words that are composed of more
than one upper case letters. Usually one would use</p>
<pre data-lang="css" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-css "><code class="language-css" data-lang="css"><span style="color:#9b703f;">.caps </span><span>{ text-variant: </span><span style="color:#949175;">small-caps</span><span>; }
</span></code></pre>
<p>to enable a font’s small caps variant for these words. Unfortunately, the PT
Serif font does not provide small caps<sup class="footnote-reference"><a href="#1">1</a></sup> which is why I had to fake them with
a trick by [Paul LLoyd][]. He uses [PT Serif Caption][] with a font size of
0.8125em relative to the body font:</p>
<figure>
<img src="/images/2013-03-03/caps.png" />
</figure>
<p>The ipsum sample to the left uses the standard PT Serif font with a reduced size
and is a bit too skinny and narrow. The size-reduced PT Serif Caption sample to
the right has a similar amount of stroke thickness as the rest of the letters
and does not stand out too much.</p>
<h2 id="and-other-features">… and other features</h2>
<p>Besides adding small caps classes, Typogruby also</p>
<ul>
<li>avoids [widows][] by inserting an <code>&nbsp;</code> before the last word,</li>
<li>replaces special characters such as <code>"quotes"</code>, <code>...</code> and <code>---</code> with
“quotes”, … and —,</li>
<li>wraps the ampersand character & in <code>span.amp</code> for further styling,</li>
<li>and wraps the first quote in a starting line in <code>span.dquo</code> for implementing
hanging punctuation.</li>
</ul>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>PT Serif Pro has small caps. But this font is not available via Typekit. And I am not sure to spend much money on a not-for-profit blog.
[Paul Lloyd]: http://dribbble.com/shots/834370-Simulated-Small-Caps-for-PT-Serif
[PT Serif Caption]: http://www.google.com/webfonts/specimen/PT+Serif+Caption
[widows]: http://en.wikipedia.org/wiki/Widows_and_orphans</p>
</div>
Writing C in the 21st century2013-02-27T00:00:00+00:002013-02-27T00:00:00+00:00
Unknown
https://bloerg.net/posts/writing-c-in-the-21st-century/<p>The Internet Systems Consortium, or ISC for short, is well known for developing
and publishing Internet system software such as BIND and DHCP. Today, they
<a href="https://www.isc.org/wordpress/programming-languages-for-bind-10/">commented</a> on why they re-wrote BIND 10 in C++ and Python. I think this needs
some discussion. They write</p>
<blockquote>
<p>So when ISC started seriously thinking about BIND 10 – around 2006 or so – the
question of what language to use for the new project came up.</p>
<p>The first question is of course, “Why not C?” Some answers are:</p>
<p>String manipulation in C is a tedious chore</p>
</blockquote>
<p>String manipulation in C <em>can be</em> a tedious chore when you write something like
this (taken from <code>lib/isccc/cc.c</code>)</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>len </span><span style="color:#72ab00;">= </span><span style="color:#b39f04;">strlen</span><span>(_frm) </span><span style="color:#72ab00;">+ </span><span style="color:#b39f04;">strlen</span><span>(_to) </span><span style="color:#72ab00;">+ </span><span style="color:#b39f04;">strlen</span><span>(_ser) </span><span style="color:#72ab00;">+ </span><span style="color:#b39f04;">strlen</span><span>(_tim) </span><span style="color:#72ab00;">+ </span><span style="color:#b3933a;">4</span><span>;
</span><span>key </span><span style="color:#72ab00;">= </span><span style="color:#b39f04;">malloc</span><span>(len);
</span><span style="color:#72ab00;">if </span><span>(key </span><span style="color:#72ab00;">== </span><span style="color:#b3933a;">NULL</span><span>)
</span><span> </span><span style="color:#72ab00;">return </span><span>(ISC_R_NOMEMORY);
</span><span style="color:#5597d6;">snprintf</span><span>(key, len, </span><span style="color:#d07711;">"</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">;</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">;</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">;</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">"</span><span>, _frm, _to, _ser, _tim);
</span></code></pre>
<p>all the time and don’t follow the DRY rule. Not only is</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>key </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">g_strdup_printf </span><span>(</span><span style="color:#d07711;">"</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">;</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">;</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">;</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">"</span><span>, _frm, _to, _ser, _tim);
</span></code></pre>
<p>shorter and easier to grasp, it is also a lot less dangerous than calculating
the final string length by hand. Even if ISC decides not to use a portable C
library such as GLib or Apache APR, they could still write something like that
on their own instead of relying on C’s standard library.</p>
<blockquote>
<p>Error handling is optional and cumbersome</p>
</blockquote>
<p>Yes, error handling is optional and sometimes it is cumbersome but exception
handling as used in C++ or Java is no silver bullet either. Whereas
C<sup class="footnote-reference"><a href="#c-errors">1</a></sup> and <a href="http://dave.cheney.net/2012/01/18/why-go-gets-exceptions-right">Go</a> explicitly tell the developer “look, this might be a
potentially dangerous call”, a C++ program might explode in places where you
don’t expect that to happen.</p>
<blockquote>
<p>Encapsulation and other object-oriented features must be emulated</p>
</blockquote>
<p>Encapsulation must not be “emulated”, it is part of the language design.
Thousands of robust and cleanly written libraries have shown how this works for
decades now: provide an opaque data structure and let the user access it via
public API functions. I wonder what the other ominous, object-oriented features
are.</p>
<blockquote>
<p>C lacks good memory management</p>
</blockquote>
<p>What constitutes “good” memory management? Fast, predictable and extensible? Or
safe and simple with a garbage collector? Both is possible with C. On the other
hand, C++ is a lot better at <em>resource</em> management if RAII is used. But then, it
must be implemented consequently.</p>
<hr />
<p>I am not saying, that C++ is a bad choice nor that C does wonders for BIND. But,
naive and untrue statements about C and strange reasons in favor of C++ like
this</p>
<blockquote>
<p>C++ is also a very popular language, and also has all of the features we are
looking for. However, C++ is by no means an easy language to work with, so the
idea is that we will avoid its complexity when possible.</p>
</blockquote>
<p>deserves some clarification. C is [not dead][] and even today, new projects can
be written in it in a sound and safe way. But the people must be willing to
dedicate a little bit of their precious time.</p>
<p>A lively discussion about ISC’s blog post is going on in [this][] HN comment
thread.</p>
<div class="footnote-definition" id="c-errors"><sup class="footnote-definition-label">1</sup>
<p>Yes, I know you can ignore return values. But at least you know, that a function that returns an error code can fail. A C++ exception is handed up the stack to whoever catches it first.
[not dead]: http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html
[this]: http://news.ycombinator.com/item?id=5291750</p>
</div>
Integrating a GTK+ C widget into Python2013-02-23T00:00:00+00:002013-02-23T00:00:00+00:00
Unknown
https://bloerg.net/posts/integrating-gtk-c-widget-in-python/<p>Last year I wrote a <a href="/2012/07/04/a-gtk-property-tree-view.html">GTK+ widget</a> to show and modify the properties of any
<code>GObject</code>. The widget was written in C for use in a C application. For another
project, I also have to modify the properties of <code>GObject</code>s but this time the
application is written in Python using <a href="https://live.gnome.org/PyGObject">PyGObject</a>. Here’s how I did it.</p>
<h2 id="python-extension">Python extension</h2>
<p>Obviously, a widget that is written in pure C and does not come with any
introspection data is unknown to Python’s type system. For this reason, we must
write a small extension module, that instantiates the object. In my case, I want
to call <code>egg_property_tree_view_new()</code> that either takes <code>NULL</code> or an object
that is going to be observed. First we include headers for Python, Python
GObject access and our widget:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>#include <Python.h>
</span><span>#include <pygobject.h>
</span><span>#include "egg-property-tree-view.h"
</span></code></pre>
<p>Our module provides one function called <code>create_property_view</code> that takes a
GObject object and returns the constructed tree view object:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#668f14;">static</span><span> PyObject </span><span style="color:#72ab00;">*
</span><span style="color:#c23f31;">create_property_view </span><span>(PyObject </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">self</span><span>,
</span><span> PyObject </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">args</span><span>)
</span><span>{
</span><span> PyObject </span><span style="color:#72ab00;">*</span><span>object;
</span><span> GObject </span><span style="color:#72ab00;">*</span><span>observed;
</span><span> GtkWidget </span><span style="color:#72ab00;">*</span><span>widget;
</span></code></pre>
<p>First, let’s check if an argument was passed:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span> </span><span style="color:#72ab00;">if </span><span>(</span><span style="color:#72ab00;">!</span><span style="color:#5597d6;">PyArg_ParseTuple </span><span>(args, </span><span style="color:#d07711;">"O"</span><span>, </span><span style="color:#72ab00;">&</span><span>object))
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#b3933a;">NULL</span><span>;
</span></code></pre>
<p>If the argument is not <code>None</code>, we unwrap the contained object with
<code>pygobject_get()</code>:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span> observed </span><span style="color:#72ab00;">=</span><span> object </span><span style="color:#72ab00;">==</span><span> Py_None </span><span style="color:#72ab00;">? </span><span style="color:#b3933a;">NULL </span><span style="color:#72ab00;">: </span><span style="color:#5597d6;">pygobject_get </span><span>(object);
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>((observed </span><span style="color:#72ab00;">!= </span><span style="color:#b3933a;">NULL</span><span>) </span><span style="color:#72ab00;">&& </span><span>(</span><span style="color:#72ab00;">!</span><span style="color:#5597d6;">G_IS_OBJECT </span><span>(observed)))
</span><span> </span><span style="color:#72ab00;">goto</span><span> err;
</span></code></pre>
<p>If everything is okay, we create a new widget with our C API, wrap it using
<code>pygobject_new()</code> and return it:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span> widget </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">egg_property_tree_view_new </span><span>(observed);
</span><span> object </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">pygobject_new </span><span>(</span><span style="color:#5597d6;">G_OBJECT </span><span>(widget));
</span><span> </span><span style="color:#72ab00;">return</span><span> object;
</span></code></pre>
<p>If something unexpected happened, we throw a type error exception:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>err</span><span style="color:#72ab00;">:
</span><span> </span><span style="color:#5597d6;">PyErr_SetString </span><span>(PyExc_TypeError,
</span><span> </span><span style="color:#d07711;">"Argument must be a GObject"</span><span>);
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#b3933a;">NULL</span><span>;
</span><span style="background-color:#562d56bf;color:#f8f8f8;">}</span><span>
</span></code></pre>
<p>Last but not least, we define our exported function and initialize our module.
Note, that you <em>have to</em> call <code>init_pygobject()</code> for GTK+ 2 or
<code>pygobject_init()</code> for GTK+ 3, or any attempt to call <code>pygobject_foo()</code> will
fail badly.</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#668f14;">static</span><span> PyMethodDef methods[] </span><span style="color:#72ab00;">= </span><span>{
</span><span> { </span><span style="color:#d07711;">"create_property_view"</span><span>, create_property_view,
</span><span> METH_VARARGS, </span><span style="color:#d07711;">"Create EggPropertyTreeView" </span><span>},
</span><span> { </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">NULL </span><span>}
</span><span>};
</span><span>
</span><span>PyMODINIT_FUNC
</span><span style="color:#c23f31;">initfoomodule </span><span>(</span><span style="color:#668f14;">void</span><span>)
</span><span>{
</span><span> PyObject </span><span style="color:#72ab00;">*</span><span>module </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">Py_InitModule </span><span>(</span><span style="color:#d07711;">"foomodule"</span><span>, methods);
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>(module </span><span style="color:#72ab00;">== </span><span style="color:#b3933a;">NULL</span><span>)
</span><span> </span><span style="color:#72ab00;">return</span><span>;
</span><span>
</span><span> </span><span style="color:#5597d6;">init_pygobject </span><span>();
</span><span>}
</span></code></pre>
<h2 id="build-environment">Build environment</h2>
<p>Once we have our Python wrapper together, we need to build it. For distribution
purposes it is a good idea to use <code>distutils</code> for that. Unfortunately, distutils
does not come with built-in support for pkg-config. Here I used a <a href="http://www.python-forum.de/viewtopic.php?f=11&t=19167">function</a>
from the German Python forums that calls pkg-config and creates a dict<sup class="footnote-reference"><a href="#change">1</a></sup>
for consumption by <code>distutils.core.Extension</code>. The following <code>setup.py</code> builds
the module using</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">from </span><span>distutils.core </span><span style="color:#72ab00;">import </span><span>setup, Extension
</span><span style="color:#72ab00;">from </span><span>subprocess </span><span style="color:#72ab00;">import </span><span style="color:#5597d6;">PIPE</span><span>, Popen
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">pkgconfig</span><span>(</span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">packages</span><span>):
</span><span> flags </span><span style="color:#72ab00;">= </span><span>{
</span><span> </span><span style="color:#d07711;">'-D'</span><span>: </span><span style="color:#d07711;">'define_macros'</span><span>,
</span><span> </span><span style="color:#d07711;">'-I'</span><span>: </span><span style="color:#d07711;">'include_dirs'</span><span>,
</span><span> </span><span style="color:#d07711;">'-L'</span><span>: </span><span style="color:#d07711;">'library_dirs'</span><span>,
</span><span> </span><span style="color:#d07711;">'-l'</span><span>: </span><span style="color:#d07711;">'libraries'</span><span>}
</span><span> cmd </span><span style="color:#72ab00;">= </span><span>[</span><span style="color:#d07711;">'pkg-config'</span><span>, </span><span style="color:#d07711;">'--cflags'</span><span>, </span><span style="color:#d07711;">'--libs'</span><span>] </span><span style="color:#72ab00;">+ </span><span style="color:#a2a001;">list</span><span>(packages)
</span><span> proc </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">Popen</span><span>(cmd, </span><span style="color:#5597d6;">stdout</span><span style="color:#72ab00;">=</span><span style="color:#5597d6;">PIPE</span><span>, </span><span style="color:#5597d6;">stderr</span><span style="color:#72ab00;">=</span><span style="color:#5597d6;">PIPE</span><span>)
</span><span> output, error </span><span style="color:#72ab00;">= </span><span>proc.stdout.</span><span style="color:#5597d6;">read</span><span>(), proc.stderr.</span><span style="color:#5597d6;">read</span><span>()
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>error:
</span><span> </span><span style="color:#72ab00;">raise </span><span style="color:#a2a001;">ValueError</span><span>(error)
</span><span>
</span><span> config </span><span style="color:#72ab00;">= </span><span>{}
</span><span>
</span><span> </span><span style="color:#72ab00;">for </span><span>token </span><span style="color:#72ab00;">in </span><span>output.</span><span style="color:#5597d6;">split</span><span>():
</span><span> </span><span style="color:#72ab00;">if </span><span>token </span><span style="color:#72ab00;">!= </span><span style="color:#d07711;">'-pthread'</span><span>:
</span><span> flag, value </span><span style="color:#72ab00;">= </span><span>token[:</span><span style="color:#b3933a;">2</span><span>], token[</span><span style="color:#b3933a;">2</span><span>:]
</span><span> config.</span><span style="color:#5597d6;">setdefault</span><span>(flags[flag], []).</span><span style="color:#5597d6;">append</span><span>(value)
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#d07711;">'define_macros' </span><span style="color:#72ab00;">in </span><span>config:
</span><span> macros </span><span style="color:#72ab00;">= </span><span>[(name, </span><span style="color:#b3933a;">None</span><span>) </span><span style="color:#72ab00;">for </span><span>name </span><span style="color:#72ab00;">in </span><span>config[</span><span style="color:#d07711;">'define_macros'</span><span>]]
</span><span> config[</span><span style="color:#d07711;">'define_macros'</span><span>] </span><span style="color:#72ab00;">= </span><span>macros
</span><span>
</span><span> </span><span style="color:#72ab00;">return </span><span>config
</span><span>
</span><span>module </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">Extension</span><span>(</span><span style="color:#d07711;">'foomodule'</span><span>,
</span><span> </span><span style="color:#5597d6;">sources</span><span style="color:#72ab00;">=</span><span>[</span><span style="color:#d07711;">'module/egg-property-cell-renderer.c'</span><span>,
</span><span> </span><span style="color:#d07711;">'module/egg-property-tree-view.c'</span><span>,
</span><span> </span><span style="color:#d07711;">'module/foomodule.c'</span><span>],
</span><span> </span><span style="color:#5597d6;">extra_compile_args</span><span style="color:#72ab00;">=</span><span>[</span><span style="color:#d07711;">'-std=c99'</span><span>],
</span><span> </span><span style="color:#72ab00;">**</span><span style="color:#5597d6;">pkgconfig</span><span>(</span><span style="color:#d07711;">'gtk+-2.0 pygobject-2.0'</span><span>))
</span><span>
</span><span style="color:#5597d6;">setup</span><span>(
</span><span> </span><span style="color:#5597d6;">ext_modules</span><span style="color:#72ab00;">=</span><span>[module]
</span><span> </span><span style="color:#7f8989;"># Further package description
</span><span>)
</span></code></pre>
<div class="footnote-definition" id="change"><sup class="footnote-definition-label">1</sup>
<p>I had to change the code a bit, because GTK+ requires the <code>-pthread</code> cflag, on which the token “parser” chokes on.</p>
</div>
<h2 id="glueing-everything-together">Glueing everything together</h2>
<p>Now,</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>python setup.py install
</span></code></pre>
<p>builds our extension module and installs it into the global <code>site-packages</code> or
– as I usually do – into a virtualenv. With everything in place, we can
happily mix and match objects created via GObject introspection or our own
wrappers:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">import </span><span>foomodule
</span><span style="color:#72ab00;">from </span><span>gi.repository </span><span style="color:#72ab00;">import </span><span>Gtk
</span><span>
</span><span>window </span><span style="color:#72ab00;">= </span><span>Gtk.</span><span style="color:#5597d6;">Window</span><span>()
</span><span>view </span><span style="color:#72ab00;">= </span><span>foomodule.</span><span style="color:#5597d6;">create_property_view</span><span>(window)
</span><span>window.</span><span style="color:#5597d6;">add</span><span>(view)
</span><span>window.</span><span style="color:#5597d6;">show_all</span><span>()
</span><span>
</span><span>Gtk.</span><span style="color:#5597d6;">main</span><span>()
</span></code></pre>
<p>There is a disadvantage of a hand-written wrapper compared to the generated
introspection data: you have to translate every C method manually. However, in
most cases, widgets are entirely characterized by their properties. Hence, a
Python programmer can use <code>object.set_properties</code> or <code>object.props</code> to change
the behaviour of the widget instead of calling
<code>object.set_certain_property(value)</code>.</p>
Refactoring code2013-02-15T00:00:00+00:002013-02-15T00:00:00+00:00
Unknown
https://bloerg.net/posts/refactoring-code/<p>When I started doing my current job, I was well aware of the benefits of writing
“good” code with clearly defined interfaces and small units of code (at any
level such as translation units, classes or functions) that doesn’t repeat
itself over and over again. But I sinned because of time constraints and general
laziness.</p>
<p>I was enlightened when I read <em>Clean Code</em> by Robert C. “Uncle Bob” Martin. In
this book, Martin explains reasons and detailed steps for refactoring
code<sup class="footnote-reference"><a href="#refactoring">1</a></sup>. Don’t get me wrong, Martin is very dogmatic, the code
examples too Java-centric and many observations are pretty obvious, but the book
gave me the right amount of motivation to care more about the stuff that I
produce.</p>
<p>Now, I was positively surprised when I stumbled upon [SourceMaking.com][], a
compilation of software methods and tools published under CreativeCommons. Yes,
it contains bits about UML and design patterns but the absolute highlight is the
[refactoring][] section. Essentially, everything that’s written in <em>Clean Code</em>
is presented in a logical and easily accessible way. For each refactoring
method, a motivation is given why this method exists, mechanics how this applies
to certain code structures and an example in Java.</p>
<p>But before diving into each refactoring method, make sure to read about the
[bad code smells][] and reflect on that. I guess every one of us, is guilty of
duplicated code and too long code sections.</p>
<div class="footnote-definition" id="refactoring"><sup class="footnote-definition-label">1</sup>
<p>Prior to this, whenever I heard the word <em>refactoring</em> I was immediately thinking about renaming classes and methods. Sure, renaming is important but not crucial to refactoring.
[SourceMaking.com]: http://sourcemaking.com
[refactoring]: http://sourcemaking.com/refactoring
[bad code smells]: http://sourcemaking.com/refactoring/bad-smells-in-code</p>
</div>
2013 Redesign2013-02-09T00:00:00+00:002013-02-09T00:00:00+00:00
Unknown
https://bloerg.net/posts/2013-redesign/<p>For some time now, I was thinking about a redesign of the bloerg website. It was
looking good but there were warts here and there and the style sheet was an unmaintable
mess. Now that I had a talk with a friend about static site generation and
design in general, I felt motivated enough to create a new style from scratch. In a
couple of hours …</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>8 files changed, 221 insertions(+), 436 deletions(-)
</span></code></pre>
<p>I tried to keep the general idea of the old page but increase legibility by
these measures:</p>
<ol>
<li>Larger font size (I’m probably getting old).</li>
<li>PT Serif instead of PT Sans as the body font and Lato for headlines.</li>
<li>Consistent use of relative <code>em</code> units throughout the site instead of a
mixture of <code>em</code>, <code>px</code> and <code>pt</code>.</li>
<li>Dumped the index with teaser texts in favor of a paginated list of all posts
and a separate <a href="/archive.html">archive</a> page.</li>
<li>Wikipedia-like footnote highlighting.</li>
</ol>
<p>There might be some glitches left. If you notice any, tell me and I will fix
them as soon as possible.</p>
<p>Otherwise, <em>farewell old page.</em></p>
Make (efficient) use of ~/.ssh/config2013-01-21T00:00:00+00:002013-01-21T00:00:00+00:00
Unknown
https://bloerg.net/posts/ssh-config/<p>Last year, a <a href="http://nerderati.com/2011/03/simplify-your-life-with-an-ssh-config-file/">blog entry</a> by Joël “from the Internet” caught my attention on
HN. He wrote about the benefits of using an SSH configuration file to avoid
typing full host names, user login names and deviating SSH ports and reduce the
work to make a port forwarded connection. Because it simplified my life as
advertised, I strongly urge you to read the blog post right now!</p>
<p>Things get even more efficient, if you have a large array of machines with host
names that follow a certain pattern. In my case, I have access to a cluster of
GPU and CPU nodes, all bearing the same prefix and a per-machine number. Instead
of repeating myself over and over again, I can specify a wild carded pattern
that matches zero or more (<code>*</code>) characters or exactly one (<code>?</code>):</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Host foo-*-?.bar.baz.co.uk
</span><span> User me
</span><span> IdentityFile ~/.ssh/foo_key
</span></code></pre>
<p>Still, it is tiresome to type the fully qualified host name. But it can be
shortened (provided that OpenSSH>=5.6 is installed) by re-using the expanded
nickname with the <code>%h</code> specifier:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>Host foo-*-?
</span><span> User me
</span><span> HostName %h.bar.baz.co.uk
</span><span> IdentityFile ~/.ssh/foo_key
</span></code></pre>
<p>Now, I can simply log into a machine with <code>ssh foo-gpu-1</code> instead of <code>ssh -i ~/.ssh/foo_key me@foo-gpu-1.bar.baz.co.uk</code>. The <a href="http://manpages.ubuntu.com/manpages/precise/en/man5/ssh_config.5.html">manpage</a> on <code>ssh_config</code> is a
nice and short read, so don’t miss out the other available options.</p>
OpenCL resource management2013-01-15T00:00:00+00:002013-01-15T00:00:00+00:00
Unknown
https://bloerg.net/posts/opencl-resource-management/<p>It is common sense that for a long-running system, it’s of utmost importance
that no piece of code is leaking memory or other resources. Otherwise, the
system will crash sooner or later due to memory exhaustion. This is what
happened with a distributed system I developed for high performance image
processing.</p>
<p>In this system, each network node – each having at least one GPU – runs a
small piece of server software that prepares an image processing graph and
processes incoming data. When the data stream has been processed, the server
releases almost all resources and waits for the next data processing task. To be
sure, that everything was working as expected, I ran a stress test during the
night. Of course, at some point the server crashed with a segmentation fault.
Using <a href="http://manpages.ubuntu.com/manpages/precise/man1/alt-nvidia-current-smi.1.html">nvidia-smi</a>, I found out that the GPU has allocated more and more
memory and faced an out-of-memory situation at some point.</p>
<p>OpenCL has a straightforward system of reference counted resources. You create a
context, command queue, buffer, program or kernel with its corresponding
<code>clCreateFoo()</code> function. If several independent software parts reference an
OpenCL resource for internal use, the reference count can be increased with
<code>clRetainFoo()</code>. When a resource is no longer of any use, it can be discarded
with <code>clReleaseFoo()</code>. Once the reference count reaches zero, any associated
resources such as memory are freed. At least, that is what I expected.</p>
<p>However on NVIDIA systems, a final <code>clReleaseMemObject()</code> will <em>not</em> free
the memory segment in GPU memory, if not <em>every</em> other OpenCL object has been
freed too. Consider this small snippet:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>n_elements </span><span style="color:#72ab00;">= </span><span style="color:#b3933a;">1024 </span><span style="color:#72ab00;">* </span><span style="color:#b3933a;">1024</span><span>;
</span><span>mem </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">clCreateBuffer </span><span>(context, CL_MEM_READ_WRITE,
</span><span> n_elements </span><span style="color:#72ab00;">* sizeof </span><span>(</span><span style="color:#668f14;">float</span><span>),
</span><span> </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#72ab00;">&</span><span>errcode);
</span><span>
</span><span style="color:#7f8989;">/* Launch kernel with one parameter */
</span><span style="color:#5597d6;">clSetKernelArg </span><span>(kernel, </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#72ab00;">sizeof </span><span>(cl_mem), </span><span style="color:#72ab00;">&</span><span>mem);
</span><span style="color:#5597d6;">clEnqueueNDRangeKernel </span><span>(cmd_queue, kernel,
</span><span> </span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#72ab00;">&</span><span>n_elements, </span><span style="color:#b3933a;">NULL</span><span>,
</span><span> </span><span style="color:#b3933a;">0</span><span>, </span><span style="color:#b3933a;">NULL</span><span>, </span><span style="color:#72ab00;">&</span><span>event);
</span><span>
</span><span style="color:#7f8989;">/* Wait for end of execution and release all resources */
</span><span style="color:#5597d6;">clWaitForEvents </span><span>(</span><span style="color:#b3933a;">1</span><span>, </span><span style="color:#72ab00;">&</span><span>event)</span><span style="background-color:#562d56bf;color:#f8f8f8;">)</span><span>;
</span><span style="color:#5597d6;">clReleaseMemObject </span><span>(mem);
</span><span style="color:#5597d6;">clReleaseKernel </span><span>(kernel);
</span><span style="color:#5597d6;">clReleaseProgram </span><span>(program);
</span><span style="color:#5597d6;">clReleaseCommandQueue </span><span>(cmd_queue);
</span><span style="color:#5597d6;">clReleaseContext </span><span>(context);
</span></code></pre>
<p>This looks innocent, however repeating this over and over again will give you a
<code>CL_MEM_OBJECT_ALLOCATION_FAILURE</code> eventually because we did not release the
<code>event</code> object and thus did not clean up GPU memory.</p>
<p>So, if you are developing with NVIDIA’s OpenCL implementation and what to ensure
a stable system:</p>
<ul>
<li>Check that each call to <code>clCreateFoo()</code> and <code>clRetainFoo()</code> is accompanied
by a <code>clReleaseFoo()</code> call.</li>
<li>Beware of <code>cl_event</code> objects that are created implicitly by calls to the
<code>clEnqueueFoo()</code> function family.</li>
<li>Do not assume that <code>clReleaseContext()</code> will release resources that are
associated with the context.</li>
</ul>
<h3 id="the-oclkit-api">The oclkit API</h3>
<p>To check the leakage behaviour, I wrote a small OpenCL application. I did this
<a href="https://github.com/matze/qperf">many</a> <a href="https://github.com/matze/ocl-regressions">times</a> <a href="https://github.com/matze/ocl-copy-vs-ref">before</a>, but until now, I always copied the boiler plate
from here to there. To avoid this in the future, I came up with a sanitized
version of the boiler plate code called <a href="https://github.com/matze/oclkit">oclkit</a>. It’s simple C99-only code
that sets up a valid OpenCL context and command queues for each device. You can
plug oclkit into your own (GPL-compatible) apps or just run the <em>leak</em> example
with <code>./configure && make && ./leak</code>.</p>
GObject Introspection Tips2012-12-17T00:00:00+00:002012-12-17T00:00:00+00:00
Unknown
https://bloerg.net/posts/gobject-introspection-tips/<p>GObject comes with a well-thought introspection <a href="https://live.gnome.org/GObjectIntrospection">mechanism</a> that helps
building language bindings by adhering to certain <a href="http://developer.gnome.org/gobject/2.30/gtype-conventions.html">conventions</a>. Unfortunately,
the documentation is lacking structure and some vital information. Here are some
tips, to help you through the initial confusion, if you plan to make your
library introspectable:</p>
<ul>
<li>
<p>If possible and to avoid a lot of trouble, you should use the Autotools
integration as described <a href="https://live.gnome.org/GObjectIntrospection/AutotoolsIntegration">here</a>. But if you have to use CMake, the
following snippet should give you a good start (assuming Foo is your library’s
namespacing prefix):</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span> find_program(INTROSPECTION_SCANNER "g-ir-scanner")
</span><span> find_program(INTROSPECTION_COMPILER "g-ir-compiler")
</span><span>
</span><span> set(GIR_PREFIX "Foo-${FOO_API_VERSION}")
</span><span> set(GIR_XML "${GIR_PREFIX}.gir")
</span><span> set(GIR_TYPELIB "${GIR_PREFIX}.typelib")
</span><span>
</span><span> add_custom_command(OUTPUT ${GIR_XML}
</span><span> COMMAND ${INTROSPECTION_SCANNER}
</span><span> --namespace=Foo
</span><span> --nsversion=${FOO_API_VERSION}
</span><span> --library=foo
</span><span> --no-libtool
</span><span> --include=GObject-2.0
</span><span> --include=GModule-2.0
</span><span> --output ${GIR_XML}
</span><span> --warn-all
</span><span> ${FOO_SRCS}
</span><span> DEPENDS ${FOO_SRCS}
</span><span> WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
</span><span>
</span><span> add_custom_command(OUTPUT ${GIR_TYPELIB}
</span><span> COMMAND ${INTROSPECTION_COMPILER}
</span><span> -o ${GIR_TYPELIB}
</span><span> ${GIR_XML}
</span><span> DEPENDS ${GIR_XML}
</span><span> WORKING_DIRECTORY
</span><span> ${CMAKE_CURRENT_BINARY_DIR})
</span><span>
</span><span> add_custom_target(gir ALL DEPENDS ${GIR_XML} ${GIR_TYPELIB})
</span><span> add_dependencies(gir foo)
</span></code></pre>
<p>Unfortunately, the typelib is not found if it is installed into
<code>/usr/local/lib/girepository-1.0</code>. You either need to install it into
<code>/usr/lib/girepository-1.0</code> or extend <code>GI_TYPELIB_PATH</code>.</p>
</li>
<li>
<p>If you write interfaces, make sure that the definitions of the “virtual”
function pointers in the interface structure match the real functions used to
call the actual implementation. In fact, the parameter names should not play a
role, but <code>g-ir-scanner</code> is very finicky. Something like this</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span> struct _FooVehicle {
</span><span> void (*accelerate) (FooVehicle *vehicle, guint kmh);
</span><span> };
</span><span>
</span><span> void foo_vehicle_accelerate (FooVehicle *vehicle, guint speed);
</span></code></pre>
<p>will give you a “Foo: Vehicle: Virtual function ‘accelerate’ has no known
invoker” although the type signature is the same!</p>
</li>
<li>
<p>Non-GObject-based libraries cannot be inspected. If the user of the bindings
does not have to care about interfacing with third-party libraries you
can just type their data structures as <code>gpointer</code> but otherwise you must wrap
the entire library.</p>
</li>
<li>
<p>You must keep object references in mind and set the <code>transfer</code> annotation
appropriately. If you see crashes at the end of a Python session, it’s very
likely that the Python interpreter tries to unref an object that has already
been destroyed by your library.</p>
</li>
<li>
<p>Be aware of translations into the target language. In C, the canonical
constructor would be something like <code>foo_vehicle_new (...)</code> or
<code>foo_vehicle_new_with_gasoline (...)</code>. In Python, <code>g_object_new</code> is used to
instantiate the object thus bypassing any code in the <code>_new</code> functions. Of
course, you could still call <code>Foo.Vehicle.new()</code> directly though.</p>
<p>If you want to make sure that certain parameters are passed at construction
time, you should add GObject properties with the <code>G_PARAM_CONSTRUCT</code> or
<code>G_PARAM_CONSTRUCT_ONLY</code> flag to your class. To initialize the object
after all “constructable” properties are set, you can override the
virtual <code>constructed</code> function of the GObject base class.</p>
</li>
<li>
<p>Always put <code>GError</code> arguments at the end of the parameter list or before the
final <code>...</code> argument list. At least with Python, you can then benefit from
automatic conversion to real Python exceptions.</p>
</li>
</ul>
Pandoc and LaTeX2012-12-05T00:00:00+00:002012-12-05T00:00:00+00:00
Unknown
https://bloerg.net/posts/pandoc-and-tex/<p>In a previous post, I <a href="/2012/11/23/getting-the-most-out-of-stock-latex.html">described</a> some basic ideas how to make the PDF output
of LaTeX a little bit prettier. Unfortunately, the text needs to be written in
plain TeX which is cumbersome at best. But why not write Markdown and use yours
truly’s favorite converter <a href="http://johnmacfarlane.net/pandoc/">Pandoc</a> to do the dirty job of generating TeX? The
best thing with this setup: Pandoc does not process raw TeX commands and
environments such as <code>\newpage</code> or <code>$$</code> but outputs them verbatim into the final
file. Thus, for most tasks you can write Markdown but if it’s necessary, you can
simply add high-quality TikZ figures or plots made with PGFPlots.</p>
<p>To integrate the stylistic changes, I prepared a TeX <a href="https://gist.github.com/4194445">template</a> for Pandoc’s
simple <a href="http://johnmacfarlane.net/pandoc/README.html#templates">template engine</a> and a Makefile. You can check out both in the
current (empty) directory with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ git clone git://gist.github.com/4194445.git .
</span><span>$ rm -rf .git
</span></code></pre>
<p>Now place any Markdown files in here and <code>make</code> a nicely laid out PDF document.</p>
<p>You might have noticed the title and author variables in the template which will
result in correspond <code>\title{}</code> and <code>\author{}</code> calls in the TeX document. To
enter this data in your Markdown file, you have to use Pandoc’s <a href="http://johnmacfarlane.net/pandoc/README.html#title-block">title block</a>
extension and put it on top of your Markdown file like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>% Multi line
</span><span> titles must begin with a leading space
</span><span>% Brian Griffin
</span><span> Glenn Quagmire
</span></code></pre>
<p>Unfortunately, there are also some caveats with Pandoc:</p>
<ul>
<li>Although Pandoc has a <code>$lang$</code> variable to be used for the document class and
<code>babel</code> package, the language is not detected reliably.</li>
<li>The bibliography feature is not working for me right now, because the Pandoc
dependency for managing bibliographies is not available for Ubuntu 12.04.</li>
<li>The template is and cannot be very portable. For example, I assume that the TikZ
package is already installed.</li>
</ul>
<p>For me, these are only minor problems and for smaller documents such as reports
I now write Markdown and convert that rather than typing the document in a word
processor or as raw LaTeX.</p>
Where is Markdown 2?2012-11-30T00:00:00+00:002012-11-30T00:00:00+00:00
Unknown
https://bloerg.net/posts/where-is-markdown-2/<p>Markdown is a success! Every developer-focused web site that I frequent supports
Markdown to enter content: Stack Overflow, GitHub, Reddit, you name it. I attribute
Markdown’s success to its syntactic simplicity yet powerful expressiveness.</p>
<figure>
<img src="/images/2012-11-30/markdown.png" alt="Markdown Interest on Google
Trends for 2004 until now."/>
<figcaption>Development of interest on Markdown over the last seven years according to
<a href="http://www.google.com/trends/explore#q=markdown">Google Trends</a>.</figcaption>
</figure>
<p>But its wide adoption led to an unsatisfying situation with sixty something
<a href="http://en.wikipedia.org/wiki/List_of_Markdown_implementations">implementations</a>, some of them deviating from John Gruber’s <a href="http://daringfireball.net/projects/markdown/">original</a>
Markdown draft with incompatible extensions. This is a serious problem. For
example, I can hardly remember all the different ways of specifying code block
languages or tables.</p>
<p>To bring the Markdown development back on track, Jeff Atwood called major
implementors<sup class="footnote-reference"><a href="#1">1</a></sup> to write a <a href="http://www.codinghorror.com/blog/2012/10/the-future-of-markdown.html">unified spec</a>. But except for causing a minor
commotion in some blog posts and tweets, nothing really happened from his side.
No spec, no follow-up post, nothing. Whatever his intentions were, he left the
chance behind to use the initial momentum for developing a Markdown
specification that could’ve succeeded all others.</p>
<p>But, not all hope is lost: For better or worse, a <a href="http://www.w3.org/community/markdown/">W3C working group</a> formed
to design a Markdown specification draft. Up to now, an impressive number of 400
mails were sent to the public <a href="http://lists.w3.org/Archives/Public/public-markdown/">mailing list</a>. But reading closer, there is
only slow progress and by now only a couple of basic Markdown elements are
covered by the provisional <a href="http://htmlpreview.github.com/?https://github.com/karlcow/markdown-testsuite/blob/master/markdown-spec.html">specification</a>. What I suspect is, that being
unsupported by major Markdown players, the spec will meet its fate as Markdown
implementation n+1.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Although he forgot to address John MiacFarlane of Pandoc fame.</p>
</div>
Getting the most out of stock LaTeX2012-11-23T00:00:00+00:002012-11-23T00:00:00+00:00
Unknown
https://bloerg.net/posts/getting-the-most-out-of-stock-latex/<p>Y’all know it: TeX and LaTeX are godsends brought to you by two CS icons and an
innumerable amount of kind people. But, you may be tired of the bland default
look from which you can tell immediately, that a document is made with TeX.
With a careful selection of packages and settings you can still a decent look
with a stock LaTeX distribution:</p>
<figure>
<img src="/images/2012-11-23/transformation.png" />
</figure>
<p>Before I’ll go into detail, a word of warning first: aesthetics is a subjective
matter and I have <em>my own</em> taste regarding text. You might not agree with this
but you still could get an idea what to change to achieve a certain look.</p>
<h2 id="the-preamble">The preamble</h2>
<p>First let’s have a look how I created the initial document:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\documentclass[a4paper,english]{scrartcl}
</span><span>\usepackage[english]{babel}
</span><span>\usepackage{hyperref}
</span><span>
</span><span>\begin{document}
</span><span>\selectlanguage{english}
</span><span>
</span><span>\section{What is a Blog}
</span><span>
</span><span>A blog (a portmanteau of the term \emph{web log}) is a discussion or
</span><span>informational site published on the
</span><span>\href{http://en.wikipedia.org/wiki/World_Wide_Web}{World Wide Web} and
</span><span>consisting of discrete entries (``posts'') typically displayed in reverse
</span><span>chronological order (the most recent post appears first). Until 2009 blogs
</span><span>were usually the work of a single individual, occasionally of a small group,
</span><span>and often were themed on a single subject. More recently MABs have
</span><span>developed, with posts written by large numbers of authors and professionally
</span><span>edited. MABs from newspapers, other media outlets, universities, think
</span><span>tanks, interest groups and similar institutions account for an increasing
</span><span>quantity of blog traffic. The rise of
</span><span>\href{http://en.wikipedia.org/wiki/Twitter}{Twitter} and other
</span><span>``\href{http://en.wikipedia.org/wiki/Microblogging}{microblogging}'' systems
</span><span>helps integrate MAB and single-author blogs into societal newstreams.
</span><span>\emph{Blog} can also be used as a verb, meaning to \emph{maintain} or
</span><span>\emph{add content} to a blog.
</span><span>
</span><span>The emergence and growth of blogs in the late 1990s coincided with the
</span><span>advent of \href{http://en.wikipedia.org/wiki/Blog_software}{web publishing
</span><span>tools} that facilitated the posting of content by non-technical users.
</span><span>(Previously, a knowledge of such technologies as
</span><span>\href{http://en.wikipedia.org/wiki/HTML}{HTML} and
</span><span>\href{http://en.wikipedia.org/wiki/FTP}{FTP} had been required to publish
</span><span>content on the Web.)
</span><span>
</span><span>\end{document}
</span></code></pre>
<p>Instead of the standard <code>article</code> document class, I typeset the Wikipedia
<a href="http://en.wikipedia.org/wiki/Blog">article</a> with the drop-in replacement from the KOMA script package. This
package provides a better overall geometry – especially compared to other
German documents – and a lot of customization opportunities. I also added the
<code>hyperref</code> package to typeset URLs. Everything looks okay so far, but not very
nice either.</p>
<h2 id="fonts">Fonts</h2>
<p>The most apparent difference between both documents is the font. Although the
default <em>Computer Modern</em> font was designed with the best intents, it was
developed by Knuth for then-common laser printers. As you can see in
<a href="http://www.levien.com/type/cmr/gain.html">this</a> comparison, it used to be a
good match on those printers but printed on a current laser printer it appears a
bit too skinny. To compensate for that, I typeset the body text with the
Palatino font. The font is available with all TeX distributions in the
<code>mathpazo</code> package and can be used with:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\usepackage{mathpazo}
</span><span>\linespread{1.05}
</span></code></pre>
<p>The increased line spacing (or leading to be typographically correct) helps to
improve the overall readability of the text.</p>
<p>By default, KOMA script already enables a sans serif font for all headlines, so
they are still typeset in a bold version of Computer Modern. Unfortunately,
this type face looks a tad too disproportionate. To fix this, I switched to the
condensed bold version of Computer Modern with:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\setkomafont{disposition}{\fontseries{sbc}\sffamily}
</span></code></pre>
<p>Your mileage may vary but I like old-style figures and real small caps, so that
figures and upper-cased words don’t stick out like sore thumbs. Both features
can be activated with the <code>osf</code> and <code>sc</code> options of the <code>mathpazo</code> package.</p>
<figure>
<img src="/images/2012-11-23/font-features.png" />
</figure>
<p>Also, the links are a little bit too obtrusive. To remove the boxes and color
the actual link text with a mellow purple, I use:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\usepackage{xcolor}
</span><span>\usepackage[colorlinks, urlcolor=purple]{hyperref}
</span></code></pre>
<h3 id="the-microtype-package">The microtype package</h3>
<p>You may wonder why we don’t just use XeTeX to setup the fonts. The long answer:
our premise is standard LaTeX. The real answer: the <a href="http://ctan.org/pkg/microtype">microtype</a> package. This
package provides <a href="http://hstuart.dk/2007/11/27/microtypography/">microtypographic</a> features such as character protrusion,
tracking and more to even out the overall greyness distribution of the text. The
most obvious benefits are less hyphenation and a reduced amount of overfull
hboxes. To enable microtype support just add the <code>microtype</code> package to the
preamble of your document and enjoy characters such as periods that now protrude
to give the impression of a more even margin:</p>
<figure>
<img src="/images/2012-11-23/protrusion.png" />
</figure>
<h2 id="indentation-vs-vertical-space">Indentation vs vertical space</h2>
<p>There are two common ways to separate two paragraphs. First we can indent the
first line of the succeeding paragraph by one <a href="http://en.wikipedia.org/wiki/En_%28typography%29">en</a> to one <a href="http://en.wikipedia.org/wiki/Em_%28typography%29">em</a> of horizontal
space. This is the prevailing style in novels and other fictional literature as
well as the default style of LaTeX. But we could also separate both paragraphs
with a vertical space resulting in the <em>block paragraph</em> style. The decision
which style to use is entirely subjective, but I tend to use block paragraphs
for technical documents.</p>
<p>To change to block paragraphs, append the <code>parskip</code> option to the <code>scrartcl</code>
class:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\documentclass[a4paper,english,parskip=half]{scrartcl}
</span></code></pre>
<p>Again, it really depends on the subject, choice of font and paper geometry
whether to use a half empty line or a full one.</p>
<h2 id="acronyms-with-small-caps">Acronyms with small caps</h2>
<p>There are a lot of acronyms in the example text for which I use the <a href="http://www.ctan.org/tex-archive/macros/latex/contrib/acronym">acronym</a>
package. This package helps you to manage the acronyms and print a summarizing
list. Just put</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\usepackage[nolist,nohyperlinks]{acronym}
</span></code></pre>
<p>into your preamble and add new acronyms with the <code>acronym</code> environment:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\begin{acronym}
</span><span> \acro{mab}{multi-author blog}
</span><span> \acro{html}{Hypertext Markup Language}
</span><span> \acro{mab}{File Transfer Protocol}
</span><span>\end{acronym}
</span></code></pre>
<p>You can insert an acronym with <code>\ac{acronym}</code> which will print the long form the
first time the macro is used, and the short form on each subsequent call. You
can always use the short form with <code>\acs{}</code> and typeset plural forms with
<code>\acp{}</code>. Now, by default the acronyms are not typeset in any special way but as
you can see I like to have them in small caps. For this we just have to redefine
the <code>\acsfont</code> macro:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>\renewcommand*{\acsfont}[1]{\textsc{#1}}
</span></code></pre>
<hr />
<p>That’s it so far and I can only encourage you to have a look at the vast amount
of packages that enhance your documents tremendously. Just to name the few that
I use regularly: booktabs, TikZ, pgfplots.</p>
CMake and distutils2012-11-10T00:00:00+00:002012-11-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/cmake-and-distutils/<p>Imagine you are writing a nice C library, together with some nice Python
bindings and build this with <a href="http://www.cmake.org">CMake</a>. Now, you would also like to deploy
pure Python packages that use the nice bindings. You could just use the
<a href="http://docs.python.org/2/library/distutils.html">distutils</a> module, hack a <code>setup.py</code> together and let the poor soul using
your software figure out how to install that. However, there are some good
reasons to let CMake build and install the package:</p>
<ul>
<li>CMake knows if a Python interpreter is installed.</li>
<li>The <code>setup.py</code> script can be configured with central information like version
numbers.</li>
<li>Each time the library changes, your package can be run and tested against it.</li>
<li>The poor soul can type <code>sudo make install</code> and forget everything else.</li>
</ul>
<p>To avoid any misunderstandings, let’s assume the following project layout</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>.
</span><span>|-- CMakeLists.txt
</span><span>|-- src
</span><span>| |-- CMakeLists.txt
</span><span>| |-- foo.c
</span><span>| `-- foo.h
</span><span>`-- python
</span><span> |-- CMakeLists.txt
</span><span> |-- foo
</span><span> | `-- __init__.py
</span><span> `-- setup.py.in
</span></code></pre>
<p>and that <code>${PACKAGE_VERSION}</code> is already defined in the top-level
<code>CMakeLists.txt</code>. <code>setup.py.in</code> contains the usual mumbo jumbo spiced with
variables that are replaced during the configure step:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">from </span><span>distutils.core </span><span style="color:#72ab00;">import </span><span>setup
</span><span>
</span><span style="color:#5597d6;">setup</span><span>(</span><span style="color:#5597d6;">name</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'foo'</span><span>,
</span><span> </span><span style="color:#5597d6;">version</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'$</span><span style="color:#aeb52b;">{PACKAGE_VERSION}</span><span style="color:#d07711;">'</span><span>,
</span><span> </span><span style="color:#5597d6;">package_dir</span><span style="color:#72ab00;">=</span><span>{ </span><span style="color:#d07711;">''</span><span>: </span><span style="color:#d07711;">'$</span><span style="color:#aeb52b;">{CMAKE_CURRENT_SOURCE_DIR}</span><span style="color:#d07711;">' </span><span>},
</span><span> </span><span style="color:#5597d6;">packages</span><span style="color:#72ab00;">=</span><span>[</span><span style="color:#d07711;">'foo'</span><span>])
</span></code></pre>
<p>Because, CMake can build from any directory, <code>setup.py</code> will be generated
nowhere near the package root directory. For this, we set the general package
root dir via the <code>package_dir</code> parameter to <code>${CMAKE_CURRENT_SOURCE_DIR}</code> (which
is <code>python/</code> in our example).</p>
<p>To build and install the Python package with CMake, <code>python/CMakeLists.txt</code>
looks something like this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>find_program(PYTHON "python")
</span><span>
</span><span>if (PYTHON)
</span><span> set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
</span><span> set(SETUP_PY "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
</span><span> set(DEPS "${CMAKE_CURRENT_SOURCE_DIR}/module/__init__.py")
</span><span> set(OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/build/timestamp")
</span><span>
</span><span> configure_file(${SETUP_PY_IN} ${SETUP_PY})
</span><span>
</span><span> add_custom_command(OUTPUT ${OUTPUT}
</span><span> COMMAND ${PYTHON} ${SETUP_PY} build
</span><span> COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
</span><span> DEPENDS ${DEPS})
</span><span>
</span><span> add_custom_target(target ALL DEPENDS ${OUTPUT})
</span><span>
</span><span> install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install)")
</span><span>endif()
</span></code></pre>
<p>What’s so special about that? Well, the target that we add to the build system
needs a way to tell when it’s time to run. For this, the <a href="http://www.cmake.org/cmake/help/cmake2.6docs.html#command:add_custom_command">add_custom_command</a>
specifies one or more outputs on which the target depends. But the output from
<code>python setup.py build</code> is not a good candidate because it is placed in
architecture- and platform-specific directories like <code>lib.linux-x86_64-2.6</code>. And
you don’t want to test on that in a sane and portable way.</p>
<p>What we <em>can</em> do is to generate a file in the build root, each time our source
dependencies change. The simplest way to do this, is to <code>touch</code> a stamp file and
thus update its modification time. The dependency graph then looks like this:</p>
<p>{% ditaa %}
+––––+ depends +—––+ depends +––––+
| Target |———>| Stamp |———>| Source |
+––––+ +—––+ +––––+
{% endditaa %}</p>
<p>Last but not least, we install the package with the <code>CODE</code> form of the
<a href="http://www.cmake.org/cmake/help/cmake2.6docs.html#command:install">install</a> command. <code>install(CODE)</code> is tied to the install target and expects a
CMake command. For our purposes this is <code>execute_process</code> that in turn runs
<code>python setup.py install</code>.</p>
Presenting Books2012-11-01T00:00:00+00:002012-11-01T00:00:00+00:00
Unknown
https://bloerg.net/posts/presenting-books/<p>I am an avid reader, reading everything from stupid comments on <a href="http://www.heise.de">heise.de</a> to
<a href="http://en.wikipedia.org/wiki/Thomas_Mann">classics</a> of western literature and recent papers on GPGPU. When I heard
about the Amazon Paperwhite and its increased contrast and lighting, I was all
ears and researched about e-books and e-readers. I finally came to the
conclusion, that an e-reader would be a perfect device for reading text-heavy
books during my daily commute. So, despite the current lack of devices in
Germany, I pre-ordered a <a href="http://www.kobo.com/koboglo">Kobo Glo</a>.</p>
<p>Projecting my reading habits into the future, it became reasonable to think
about how to manage my purchases. I am aware of Calibre and FBReader but
honestly, I cannot stand their user interfaces. They are very good tools for
managing e-books (especially Calibre) but they stick out like a sore thumb on
the GNOME desktop due to Qt and a some weird design decisions. Consequently, I
began working on a <em>simple</em> e-book <a href="http://matze.github.com/books">application</a> targeting GNOME exclusively:</p>
<figure>
<img src="/images/2012-11-01/books.png" />
</figure>
<p>As you can see, it uses standard widgets to visualize the book database, that is
backed by sqlite, and WebKit to show the content of a book. Because I think this
application could be useful for more people than just myself, I went full force
this time and setup some development infrastructure:</p>
<ul>
<li>The build process is managed by autoconf, automake, intltool & Co. Although it
still is a nightmarish conglomerate of shell and m4 scripts, it works
surprisingly well for a lof of tasks.</li>
<li><a href="https://travis-ci.org/matze/books">Travis CI</a> monitors the build status. This <em>free</em> service is invaluable and
very simple to set up. I encourage everyone to check it out for their own
open source software. In the future, continuous integration will be even more
important when the first unit tests are written.</li>
<li>Translations are managed with a free-for-all group at <a href="https://www.transifex.com/projects/p/books/">Transifex</a>. Anyone
with an account can contribute to the translations or begin translating
languages that are not yet covered.</li>
<li>The <a href="http://matze.github.com/books/">landing page</a> was created with the help of GitHub’s automatic pages
generator.</li>
</ul>
<h2 id="installation">Installation</h2>
<p>If you fetched the sources from GitHub you need to run <code>./autogen.sh</code> first.
For this, you will need the autotools, autopoint and intltool:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ sudo apt-get install intltool autopoint libtool
</span><span>$ ./autogen.sh
</span></code></pre>
<p>After doing this or extracting from a release tarball, you need to install the
dependencies:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ sudo apt-get install libarchive-dev libsqlite3-dev libwebkitgtk-3.0-dev libgtk-3-dev libxml2-dev
</span></code></pre>
<p>and do the standard dance:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ ./configure
</span><span>$ make
</span><span>$ sudo make install
</span></code></pre>
<h2 id="the-future">The Future</h2>
<p>Although I am quite satisfied with the current status, there are still some
features missing for the magical one-dot-zero release:</p>
<ul>
<li>Synchronization across a wide range of e-reader devices. But for now the Kobo
Glo – that needs to be shipped – is my personal goal.</li>
<li>Fetching free books from various internet sources such as Project Gutenberg.</li>
<li>Fixing meta data using an open database. It is a shame how publishers fill
out meta data in their EPUB files.</li>
<li>Disabling CSS styles inserted by the publishers or overriding them with
custom styles.</li>
<li>Better integration with <a href="http://projects.gnome.org/evince/">Evince</a> that makes up for a great PDF and CBZ
viewer.</li>
<li>A plugin mechanism based on libpeas to hook into future editing and conversion
facilities.</li>
</ul>
<hr />
<p>If you find this little application useful, leave me some feedback. Moreover
don’t hesitate to report bugs and issues or contribute code or translations.</p>
A Sorted and Filtered GtkTreeView2012-10-23T00:00:00+00:002012-10-23T00:00:00+00:00
Unknown
https://bloerg.net/posts/sorted-and-filtered-tree-view/<p>For a project yet-to-be-announced, I needed a <a href="http://developer.gnome.org/gtk/stable/GtkTreeView.html">GtkTreeView</a> that can sort the
underlying data by clicking on the column header and filter out rows satisfying
a certain condition. Both problems can be solved independently rather quickly
but it took me some time and a hint on the Gtk+ mailing list to implement both
<em>at the same time</em>. For your pleasure and my fading brain activity, I will
outline the steps necessary to implement a sortable and filtered tree view in C.</p>
<p>First of all, we need a multi-columned data store to keep our data. For this
example let’s assume a database that stores articles and their corresponding
prices. We want to sort them by their name and price and we also want to filter
out all articles that exceed a certain price. For this, we define symbols to
access the columns by name rather than a magic number:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#668f14;">typedef struct
</span><span>{
</span><span> GtkListStore </span><span style="color:#72ab00;">*</span><span>articles;
</span><span> GtkTreeModelSort </span><span style="color:#72ab00;">*</span><span>sorted;
</span><span> GtkTreeModelFilter </span><span style="color:#72ab00;">*</span><span>filtered;
</span><span> gdouble max_price;
</span><span>} </span><span style="color:#c23f31;">Store</span><span>;
</span><span>
</span><span style="color:#668f14;">enum
</span><span>{
</span><span> COLUMN_ARTICLE </span><span style="color:#72ab00;">= </span><span style="color:#b3933a;">0</span><span>,
</span><span> COLUMN_PRICE,
</span><span> N_COLUMNS
</span><span>};
</span></code></pre>
<p>We create the store with <code>gtk_list_store_new</code> and add some rows:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>GtkTreeIter iter;
</span><span>Store </span><span style="color:#72ab00;">*</span><span>store;
</span><span>
</span><span>store </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">g_new0 </span><span>(Store, </span><span style="color:#b3933a;">1</span><span>);
</span><span>store-></span><span style="color:#5597d6;">articles </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_list_store_new </span><span>(N_COLUMNS,
</span><span> G_TYPE_STRING,
</span><span> G_TYPE_DOUBLE);
</span><span>
</span><span style="color:#5597d6;">gtk_list_store_append </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>, </span><span style="color:#72ab00;">&</span><span>iter);
</span><span style="color:#5597d6;">gtk_list_store_set </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>, </span><span style="color:#72ab00;">&</span><span>iter,
</span><span> COLUMN_ARTICLE, </span><span style="color:#d07711;">"Spam"</span><span>,
</span><span> COLUMN_PRICE, </span><span style="color:#b3933a;">1.20</span><span>, </span><span style="color:#72ab00;">-</span><span style="color:#b3933a;">1</span><span>);
</span><span>
</span><span style="color:#5597d6;">gtk_list_store_append </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>, </span><span style="color:#72ab00;">&</span><span>iter);
</span><span style="color:#5597d6;">gtk_list_store_set </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>, </span><span style="color:#72ab00;">&</span><span>iter,
</span><span> COLUMN_ARTICLE, </span><span style="color:#d07711;">"Beer"</span><span>,
</span><span> COLUMN_PRICE, </span><span style="color:#b3933a;">5.99</span><span>, </span><span style="color:#72ab00;">-</span><span style="color:#b3933a;">1</span><span>);
</span><span>
</span><span style="color:#5597d6;">gtk_list_store_append </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>, </span><span style="color:#72ab00;">&</span><span>iter);
</span><span style="color:#5597d6;">gtk_list_store_set </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>, </span><span style="color:#72ab00;">&</span><span>iter,
</span><span> COLUMN_ARTICLE, </span><span style="color:#d07711;">"Chewing Gum"</span><span>,
</span><span> COLUMN_PRICE, </span><span style="color:#b3933a;">0.99</span><span>, </span><span style="color:#72ab00;">-</span><span style="color:#b3933a;">1</span><span>);
</span></code></pre>
<p>By itself, a <code>GtkListStore</code> can not be filtered and sorted. However, the tree
model subclasses <a href="http://developer.gnome.org/gtk/stable/GtkTreeModelFilter.html">GtkTreeModelFilter</a> and <a href="http://developer.gnome.org/gtk/stable/GtkTreeModelSort.html">GtkTreeModelSort</a> wrap a child
model that export a reduced and sorted data set. In order to implement both
behaviours, the <code>articles</code> model becomes the child of the filtered model and the
filtered model the child of the sorted model:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>store-></span><span style="color:#5597d6;">filtered </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">GTK_TREE_MODEL_FILTER </span><span>(</span><span style="color:#5597d6;">gtk_tree_model_filter_new </span><span>(</span><span style="color:#5597d6;">GTK_TREE_MODEL </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>), </span><span style="color:#b3933a;">NULL</span><span>));
</span><span>store-></span><span style="color:#5597d6;">sorted </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">GTK_TREE_MODEL_SORT </span><span>(</span><span style="color:#5597d6;">gtk_tree_model_sort_new_with_model </span><span>(</span><span style="color:#5597d6;">GTK_TREE_MODEL </span><span>(store-></span><span style="color:#5597d6;">filtered</span><span>)));
</span></code></pre>
<p>The filtered tree model does not know which rule is in place to filter out rows.
For this example, we set a filter callback function</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#5597d6;">gtk_tree_model_filter_set_visible_func </span><span>(store-></span><span style="color:#5597d6;">filtered</span><span>,
</span><span> (GtkTreeModelFilterVisibleFunc) row_visible,
</span><span> store, </span><span style="color:#b3933a;">NULL</span><span>);
</span></code></pre>
<p>that returns <code>TRUE</code> for all rows that exceed the maximum price:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#668f14;">static</span><span> gboolean
</span><span style="color:#c23f31;">row_visible </span><span>(GtkTreeModel </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">model</span><span>,
</span><span> GtkTreeIter </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">iter</span><span>,
</span><span> Store </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">store</span><span>)
</span><span>{
</span><span> gdouble price;
</span><span>
</span><span> </span><span style="color:#5597d6;">gtk_tree_model_get </span><span>(model, iter, COLUMN_PRICE, </span><span style="color:#72ab00;">&</span><span>price, </span><span style="color:#72ab00;">-</span><span style="color:#b3933a;">1</span><span>);
</span><span> </span><span style="color:#72ab00;">return</span><span> price </span><span style="color:#72ab00;"><=</span><span> store-></span><span style="color:#5597d6;">max_price</span><span>;
</span><span>}
</span></code></pre>
<p>Our data model is ready. Now, we need to create a <code>GtkTreeView</code> for the <em>sorted</em>
model to show it to the user:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>GtkTreeView </span><span style="color:#72ab00;">*</span><span>view;
</span><span>GtkTreeModel </span><span style="color:#72ab00;">*</span><span>model;
</span><span>
</span><span>model </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">GTK_TREE_MODEL </span><span>(store-></span><span style="color:#5597d6;">sorted</span><span>);
</span><span>view </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">GTK_TREE_VIEW </span><span>(</span><span style="color:#5597d6;">gtk_tree_view_new_with_model </span><span>(model));
</span></code></pre>
<p>To define which columns we want to show, we create <code>GtkTreeViewColumn</code> objects
and add them to the view. A <code>GtkCellRenderer</code> is responsible for how a
<code>GtkTreeViewColumn</code> is presented. In our case, we create a simple text renderer
for the article and price column:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>GtkCellRenderer </span><span style="color:#72ab00;">*</span><span>renderer;
</span><span>GtkTreeViewColumn </span><span style="color:#72ab00;">*</span><span>article_column;
</span><span>GtkTreeViewColumn </span><span style="color:#72ab00;">*</span><span>price_column;
</span><span>
</span><span>renderer </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_cell_renderer_text_new </span><span>();
</span><span>
</span><span>article_column </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_tree_view_column_new_with_attributes </span><span>(
</span><span> </span><span style="color:#d07711;">"Article"</span><span>, renderer,
</span><span> </span><span style="color:#d07711;">"text"</span><span>, COLUMN_ARTICLE,
</span><span> </span><span style="color:#b3933a;">NULL</span><span>);
</span><span>
</span><span style="color:#5597d6;">gtk_tree_view_append_column </span><span>(view, article_column);
</span><span>
</span><span>price_column </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_tree_view_column_new_with_attributes </span><span>(
</span><span> </span><span style="color:#d07711;">"Price"</span><span>, renderer,
</span><span> </span><span style="color:#d07711;">"text"</span><span>, COLUMN_PRICE,
</span><span> </span><span style="color:#b3933a;">NULL</span><span>);
</span><span>
</span><span style="color:#5597d6;">gtk_tree_view_append_column </span><span>(view, price_column);
</span></code></pre>
<p>Now, this is what we have got so far:</p>
<figure>
<img src="/images/2012-10-23/test-1.png" />
</figure>
<p>Howerver, we don’t just want to present the data but also let the user interact
with it. To implement the well-known double click on a row paradigm, we react on
the <code>"row-activated"</code> signal of the view:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#5597d6;">g_signal_connect </span><span>(view, </span><span style="color:#d07711;">"row-activated"</span><span>,
</span><span> </span><span style="color:#5597d6;">G_CALLBACK </span><span>(on_row_activated), store);
</span></code></pre>
<p>Now, what happens when a user clicks on a row that has been shuffled around
because of all the filtering and sorting? Using the <code>GtkTreeIter</code> on our full,
we will probably select the wrong row. Thus, we have to undo the path conversion
that was introduced by the filter and sort model:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#668f14;">static void
</span><span style="color:#c23f31;">on_row_activated </span><span>(GtkTreeView </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">view</span><span>,
</span><span> GtkTreePath </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">path</span><span>,
</span><span> GtkTreeViewColumn </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">col</span><span>,
</span><span> Store </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">store</span><span>)
</span><span>{
</span><span> GtkTreeIter iter;
</span><span> GtkTreePath </span><span style="color:#72ab00;">*</span><span>filtered_path;
</span><span> GtkTreePath </span><span style="color:#72ab00;">*</span><span>true_path;
</span><span>
</span><span> </span><span style="color:#7f8989;">/*
</span><span style="color:#7f8989;"> * We have a path that is filtered first and then sorted. So first, let's
</span><span style="color:#7f8989;"> * undo the sort.
</span><span style="color:#7f8989;"> */
</span><span> filtered_path </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_tree_model_sort_convert_path_to_child_path </span><span>(</span><span style="color:#5597d6;">GTK_TREE_MODEL_SORT </span><span>(store-></span><span style="color:#5597d6;">sorted</span><span>),
</span><span> path);
</span><span>
</span><span> </span><span style="color:#7f8989;">/*
</span><span style="color:#7f8989;"> * Then we undo the filter and have the path to the correct row.
</span><span style="color:#7f8989;"> */
</span><span> true_path </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_tree_model_filter_convert_path_to_child_path </span><span>(</span><span style="color:#5597d6;">GTK_TREE_MODEL_FILTER </span><span>(store-></span><span style="color:#5597d6;">filtered</span><span>),
</span><span> filtered_path);
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>(</span><span style="color:#5597d6;">gtk_tree_model_get_iter </span><span>(</span><span style="color:#5597d6;">GTK_TREE_MODEL </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>), </span><span style="color:#72ab00;">&</span><span>iter, true_path)) {
</span><span> gchar </span><span style="color:#72ab00;">*</span><span>article;
</span><span> gdouble price;
</span><span>
</span><span> </span><span style="color:#5597d6;">gtk_tree_model_get </span><span>(</span><span style="color:#5597d6;">GTK_TREE_MODEL </span><span>(store-></span><span style="color:#5597d6;">articles</span><span>), </span><span style="color:#72ab00;">&</span><span>iter,
</span><span> COLUMN_ARTICLE, </span><span style="color:#72ab00;">&</span><span>article,
</span><span> COLUMN_PRICE, </span><span style="color:#72ab00;">&</span><span>price,
</span><span> </span><span style="color:#72ab00;">-</span><span style="color:#b3933a;">1</span><span>);
</span><span>
</span><span> </span><span style="color:#5597d6;">g_print </span><span>(</span><span style="color:#d07711;">"You want to buy </span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;"> for </span><span style="color:#aeb52b;">%f</span><span style="color:#d07711;">?</span><span style="color:#aeb52b;">\n</span><span style="color:#d07711;">"</span><span>, article, price);
</span><span> </span><span style="color:#5597d6;">g_free </span><span>(article);
</span><span> }
</span><span>}
</span></code></pre>
<p>We also specify which columns can be sorted and according to what model column
they are sorted (yes, the column that is shown is not necessarily the same that
is used to sort the rows):</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#5597d6;">gtk_tree_view_column_set_sort_column_id </span><span>(article_column, COLUMN_ARTICLE);
</span><span style="color:#5597d6;">gtk_tree_view_column_set_sort_column_id </span><span>(price_column, COLUMN_PRICE);
</span></code></pre>
<p>Now, we hook everything up and put the view in a window. We also add a spin
button to change the currently set maximum price:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>GtkWidget </span><span style="color:#72ab00;">*</span><span>window;
</span><span>GtkWidget </span><span style="color:#72ab00;">*</span><span>box;
</span><span>GtkWidget </span><span style="color:#72ab00;">*</span><span>spinbutton;
</span><span>GtkAdjustment </span><span style="color:#72ab00;">*</span><span>max_price;
</span><span>
</span><span>window </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_window_new </span><span>(GTK_WINDOW_TOPLEVEL);
</span><span>box </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_box_new </span><span>(GTK_ORIENTATION_VERTICAL, </span><span style="color:#b3933a;">0</span><span>);
</span><span>
</span><span>max_price </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_adjustment_new </span><span>(</span><span style="color:#b3933a;">10.99</span><span>, </span><span style="color:#b3933a;">0.01</span><span>, </span><span style="color:#b3933a;">1024.0</span><span>, </span><span style="color:#b3933a;">0.01</span><span>, </span><span style="color:#b3933a;">1.0</span><span>, </span><span style="color:#b3933a;">0.0</span><span>);
</span><span>spinbutton </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_spin_button_new </span><span>(max_price, </span><span style="color:#b3933a;">1.0</span><span>, </span><span style="color:#b3933a;">2</span><span>);
</span><span>
</span><span style="color:#5597d6;">gtk_container_add </span><span>(</span><span style="color:#5597d6;">GTK_CONTAINER </span><span>(window), box);
</span><span style="color:#5597d6;">gtk_container_add </span><span>(</span><span style="color:#5597d6;">GTK_CONTAINER </span><span>(box), view);
</span><span style="color:#5597d6;">gtk_container_add </span><span>(</span><span style="color:#5597d6;">GTK_CONTAINER </span><span>(box), spinbutton);
</span><span>
</span><span style="color:#5597d6;">g_signal_connect </span><span>(</span><span style="color:#5597d6;">G_OBJECT </span><span>(window), </span><span style="color:#d07711;">"delete-event"</span><span>,
</span><span> </span><span style="color:#5597d6;">G_CALLBACK </span><span>(gtk_main_quit), </span><span style="color:#b3933a;">NULL</span><span>);
</span><span>
</span><span style="color:#5597d6;">g_signal_connect </span><span>(</span><span style="color:#5597d6;">G_OBJECT </span><span>(max_price), </span><span style="color:#d07711;">"value-changed"</span><span>,
</span><span> </span><span style="color:#5597d6;">G_CALLBACK </span><span>(on_max_price_changed), store);
</span></code></pre>
<p>Whenever we change the price, we need to set the corresponding variable and
filter our list again:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span style="color:#668f14;">static void
</span><span style="color:#c23f31;">on_max_price_changed </span><span>(GtkAdjustment </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">adjustment</span><span>,
</span><span> Store </span><span style="color:#72ab00;">*</span><span style="color:#5597d6;">store</span><span>)
</span><span>{
</span><span> store-></span><span style="color:#5597d6;">max_price </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_adjustment_get_value </span><span>(adjustment);
</span><span> </span><span style="color:#5597d6;">gtk_tree_model_filter_refilter </span><span>(store-></span><span style="color:#5597d6;">filtered</span><span>);
</span><span>}
</span></code></pre>
<p>That’s it and these are the fruits of our labour: A filtered model with a
reduced maximum price</p>
<figure>
<img src="/images/2012-10-23/test-2.png" />
</figure>
<p>and the output after interacting with the filtered and sorted model:</p>
<figure>
<img src="/images/2012-10-23/test-3.png" />
</figure>
<p>You can compile the <a href="https://gist.github.com/3933977">complete example</a> with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>gcc `pkg-config --cflags gtk+-3.0` sorted-and-filtered-tree-view.c `pkg-config --libs gtk+-3.0`
</span></code></pre>
<p><em>Update</em>: The reason why we have two calls to <code>pkg-config</code> is that most Gtk+
symbols are not referenced during the compilation step but need to at link
stage. Thus we have to append the link flags rather than using them in front for
the compiler.</p>
<p>If there are any doubts left, consult the excellent documentation for <a href="http://developer.gnome.org/gtk/stable/TreeWidget.html">Tree
and List Widgets</a> on the gnome.org website.</p>
Ranger File Manager2012-10-17T00:00:00+00:002012-10-17T00:00:00+00:00
Unknown
https://bloerg.net/posts/ranger-file-manager/<p>In my <a href="/2012/10/03/the-default-shell.html">last post</a>, I briefly mentioned
<a href="https://github.com/joelthelion/autojump">autojump</a>’s awesome <code>j</code> command to
quickly move between directories. But once in a while you need that little extra
bang when working with files and directories. Well, there are file managers like
Midnight Commander, for those stuck in an era when people shared Monkey Island
on floppy disks but it’s certainly not suited for vi afficionados. For those,
there is <a href="http://ranger.nongnu.org/">ranger</a> – “a file manager with VI key
bindings. It provides a minimalistic and nice curses interface with a view on
the directory hierarchy. The secondary task of ranger is to psychically guess
which program you want to use for opening particular files.”</p>
<figure>
<img src="/images/2012-10-17/ranger.png" alt="ranger previewing Ruby code"/>
<figcaption>ranger previewing a Ruby source file</figcaption>
</figure>
<p>Ranger is packaged for most distribution but you can also install a Git
<a href="https://github.com/hut/ranger">checkout</a> with <code>sudo make install</code> or just run
it directly from the source tree. Just like Vim itself, ranger can be
intimidating, even more so when considering that you are dealing with files and
not text buffers on which changes can be reverted with <kbd>u</kbd>. However,
whenever in doubt just type <kbd>?</kbd> to read ranger’s manpage.</p>
<p>After starting it with <code>ranger</code>, the first thing you will notice is a window
layout similar to MacOS’ finder: The left column shows the parent directory, the
middle column the content and selection of the current directory and the right
column child directories and files or a preview of the currently selected file.</p>
<h2 id="general-usage">General usage</h2>
<p>You can navigate through the file system with either <kbd>h</kbd>, <kbd>j</kbd>,
<kbd>k</kbd> and <kbd>l</kbd> or the arrow keys. To quickly browse large lists
you can also use the well-known <kbd>g</kbd><kbd>g</kbd>, <kbd>G</kbd>,
<kbd>Ctrl</kbd>+<kbd>d</kbd> and <kbd>Ctrl</kbd>+<kbd>u</kbd> key bindings. Once
you are on a file and go left, you open it with whatever ranger thinks is
appropriate. One important key map to remember is <kbd>Ctrl</kbd>+<kbd>h</kbd>,
that shows and hides hidden dot files. You will thank me for this.</p>
<p>By default, ranger maps several locations to the <kbd>g</kbd> key binding:
<kbd>g</kbd><kbd>r</kbd> brings you to <code>/</code>, <kbd>g</kbd><kbd>e</kbd> to <code>/etc</code>,
<kbd>g</kbd><kbd>h</kbd> to <code>$HOME</code> etc. You can define more of these by
bookmarking the current directory with <kbd>m</kbd> and change to it with the
backtick key. For example, I like to bookmark <code>~/Downloads</code> with
<kbd>m</kbd><kbd>d</kbd> to the <code>d</code> register. Another quick way to descend into
the hierarchy is the <code>:find</code> tool that you can activate with <kbd>f</kbd>. Type
some letters and it will change to the first directory or open the first file
that matches the prefix <em>exactly</em>.</p>
<p>One of the greatest features of ranger, is its preview capability. However, it
relies on external programs and out of the box, ranger displays only text files
as … plain text. Therefore, you should immediately do a</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo apt-get install highlight atool caca-utils w3m poppler-utils
</span></code></pre>
<p>on Debian systems to install programs for previewing source code, HTML files,
images, archives and PDF files.</p>
<h3 id="selecting-and-filtering-files">Selecting and filtering files</h3>
<p>To modify files (from now on the term file also includes directories), you need
to select them with <kbd>Space</kbd>. If nothing is actively selected, the file
under the current cursor position is selected. A special selection key is
<kbd>v</kbd>, that selects all files that are currently not selected (including
the one under the cursor) and un-selects those that are. To quickly un-select
<em>all</em> selected files, use <kbd>V</kbd>.</p>
<p>To get a better view of a potentially large list of unrelated files, you can
filter the view with the <code>:filter</code> command (accessible through
<kbd>z</kbd><kbd>f</kbd>) that executes a simple
<a href="http://pubs.opengroup.org/onlinepubs/009695399/functions/strstr.html">strstr()</a>
search. To view all files again, run <code>:filter</code> without any arguments.</p>
<p>Another way to restrict the files without hiding them is to search for them with
<kbd>/</kbd> and navigate through the result with <kbd>n</kbd> and <kbd>N</kbd>.</p>
<h3 id="modifying-files">Modifying files</h3>
<p>With files selected, you can use ranger’s command line accessible with
<kbd>:</kbd> to <code>:delete</code>, <code>:rename</code> and <code>:chmod</code> the permissions of the
selected files. Most shell commands are also available as ranger commands, so
you don’t need to leave ranger to <code>:touch</code> a file or <code>:mkdir</code> a new directory.
However, if you want to quickly access a shell you can type <kbd>S</kbd>.</p>
<p>Moving files works similar to yanking and pasting in vi, but not exactly the
same. You usually don’t yank into registers but really mark files to be moved.
Yanked files For example, <kbd>y</kbd><kbd>y</kbd> yanks the current file
whereas <nobr><kbd>y</kbd><kbd>a</kbd></nobr> adds the current file to the list
of already yanked files. You can remove files from the yank list with
<nobr><kbd>y</kbd><kbd>r</kbd></nobr>. Pasting a yanked file somewhere else
with <nobr><kbd>p</kbd><kbd>p</kbd></nobr> copies it there, pasting a deleted
file using <kbd>d</kbd> moves it.</p>
<h2 id="customizing-ranger">Customizing ranger</h2>
<p>To make any customizations to ranger, you should first copy the configuration
files to <code>~/.config/ranger</code> with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>ranger --copy-config=all
</span></code></pre>
<p>and edit these files locally. In the directory, you will find five configuration
and customization files: 1) <code>apps.py</code> defines applications to launch for certain
file types, 2) <code>commands.py</code> defines commands to be executed in rangers command
line mode, 3) <code>options.py</code> is ranger’s actual configuration, 4) <code>rc.conf</code>
contains simple configuration such as key maps and 5) <code>scope.sh</code> defines
applications to preview certain file types.</p>
<p>In principle, everything is extremely well documented and you should be able to
figure out how to add new key maps or commands. But for the lazy ones, I will
give a quick tour.</p>
<h3 id="binding-keys">Binding keys</h3>
<p>All default key maps are defined in <code>~/.config/ranger/rc.conf</code>. Usually, I don’t
customize key maps too heavily but in this case I like to have <kbd>F</kbd>
unset the currently set filter:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>map F filter
</span></code></pre>
<p>If you want to wait for user input, you would map to the <code>console</code></p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>map F console filter pdf
</span></code></pre>
<p>and see <code>:filter pdf</code> upon pressing <kbd>F</kbd>.</p>
<h3 id="new-commands">New commands</h3>
<p>Commands are subclasses of the <code>ranger.api.commands.Command</code> class and in most
cases you will just implement the <code>execute()</code> method. In this first example, I
adapted the <code>:rename</code> command to rename underscored names to dashed names and
vice versa.</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">class </span><span style="color:#c23f31;">toggled</span><span>(</span><span style="font-style:italic;color:#b06936;">Command</span><span>):
</span><span> </span><span style="color:#7f8989;">"""
</span><span style="color:#7f8989;"> :toggled
</span><span style="color:#7f8989;">
</span><span style="color:#7f8989;"> Changes the name of currently highlighted files from foo_bar_baz.txt to
</span><span style="color:#7f8989;"> foo-bar-baz.txt and vice versa.
</span><span style="color:#7f8989;"> """
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">execute</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> </span><span style="color:#72ab00;">from </span><span>ranger.fsobject </span><span style="color:#72ab00;">import </span><span>File
</span><span> </span><span style="color:#72ab00;">from </span><span>os </span><span style="color:#72ab00;">import </span><span>access
</span><span>
</span><span> </span><span style="color:#7f8989;"># yes, this is pathetic
</span><span> s </span><span style="color:#72ab00;">= </span><span style="color:#a2a001;">str</span><span>(</span><span style="color:#5597d6;">self</span><span>.fm.env.cf)
</span><span> s1 </span><span style="color:#72ab00;">= </span><span>s.</span><span style="color:#5597d6;">replace</span><span>(</span><span style="color:#d07711;">'-'</span><span>, </span><span style="color:#d07711;">'_'</span><span>)
</span><span> s2 </span><span style="color:#72ab00;">= </span><span>s.</span><span style="color:#5597d6;">replace</span><span>(</span><span style="color:#d07711;">'_'</span><span>, </span><span style="color:#d07711;">'-'</span><span>)
</span><span> new_name </span><span style="color:#72ab00;">= </span><span style="color:#d07711;">''</span><span>.</span><span style="color:#5597d6;">join</span><span>([b </span><span style="color:#72ab00;">if </span><span>a </span><span style="color:#72ab00;">!= </span><span>b </span><span style="color:#72ab00;">else </span><span>c </span><span style="color:#72ab00;">for </span><span>(a,b,c) </span><span style="color:#72ab00;">in </span><span style="color:#b39f04;">zip</span><span>(s,s1,s2)])
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#5597d6;">access</span><span>(new_name, os.</span><span style="color:#5597d6;">F_OK</span><span>):
</span><span> </span><span style="color:#72ab00;">return
</span><span>
</span><span> </span><span style="color:#5597d6;">self</span><span>.fm.</span><span style="color:#5597d6;">rename</span><span>(</span><span style="color:#5597d6;">self</span><span>.fm.env.cf, new_name)
</span><span> f </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">File</span><span>(new_name)
</span><span> </span><span style="color:#5597d6;">self</span><span>.fm.env.cwd.pointed_obj </span><span style="color:#72ab00;">= </span><span>f
</span><span> </span><span style="color:#5597d6;">self</span><span>.fm.env.cf </span><span style="color:#72ab00;">= </span><span>f
</span></code></pre>
<p>The next example shows that you don’t have to be bothered with files at all. If
passed a command name it will print the corresponding docstring. Not too useful
actually but it demonstrates the ease of customization.</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">class </span><span style="color:#c23f31;">cmdhelp</span><span>(</span><span style="font-style:italic;color:#b06936;">Command</span><span>):
</span><span> </span><span style="color:#7f8989;">"""
</span><span style="color:#7f8989;"> :cmdhelp <command>
</span><span style="color:#7f8989;">
</span><span style="color:#7f8989;"> Show docstring of <command>
</span><span style="color:#7f8989;"> """
</span><span>
</span><span> </span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">execute</span><span>(</span><span style="color:#5597d6;">self</span><span>):
</span><span> </span><span style="color:#72ab00;">from </span><span>sys </span><span style="color:#72ab00;">import </span><span>modules
</span><span> </span><span style="color:#72ab00;">from </span><span>inspect </span><span style="color:#72ab00;">import </span><span>getmembers, isclass
</span><span>
</span><span> clsname </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">self</span><span>.</span><span style="color:#5597d6;">rest</span><span>(</span><span style="color:#b3933a;">1</span><span>)
</span><span> clsmembers </span><span style="color:#72ab00;">= </span><span style="color:#a2a001;">dict</span><span>(</span><span style="color:#5597d6;">getmembers</span><span>(modules[</span><span style="color:#a2a001;">__name__</span><span>], isclass))
</span><span> </span><span style="color:#72ab00;">if </span><span>clsname </span><span style="color:#72ab00;">in </span><span>clsmembers:
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#5597d6;">self</span><span>.fm.</span><span style="color:#5597d6;">notify</span><span>(</span><span style="color:#d07711;">"</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">" </span><span style="color:#72ab00;">% </span><span>clsmembers[clsname].</span><span style="color:#a2a001;">__doc__</span><span>, </span><span style="color:#5597d6;">bad</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">False</span><span>)
</span><span> </span><span style="color:#72ab00;">else</span><span>:
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#5597d6;">self</span><span>.fm.</span><span style="color:#5597d6;">notify</span><span>(</span><span style="color:#d07711;">"`</span><span style="color:#aeb52b;">%s</span><span style="color:#d07711;">' is not a command!" </span><span style="color:#72ab00;">% </span><span>clsname, </span><span style="color:#5597d6;">bad</span><span style="color:#72ab00;">=</span><span style="color:#b3933a;">True</span><span>)
</span></code></pre>
<h3 id="new-previewers">New previewers</h3>
<p>Again, adding a new previewer is pretty simple. Basically, you decide how to
check what kind of type a file is (e.g. by extension or mime type), use a tool
on that file that outputs some useful information as (colored) text and put the
call in <code>scope.sh</code>. The following example shows information about Debian
packages. However, to make it work, you need to remove the deb handling from the
archive section above.</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#72ab00;">case </span><span style="color:#d07711;">"$</span><span style="color:#acb3c2;">extension</span><span style="color:#d07711;">" </span><span style="color:#72ab00;">in
</span><span> </span><span style="color:#7f8989;"># ...
</span><span> deb</span><span style="color:#72ab00;">)
</span><span> </span><span style="color:#5597d6;">dpkg --info </span><span style="color:#d07711;">"$</span><span style="color:#acb3c2;">path</span><span style="color:#d07711;">" </span><span style="color:#72ab00;">| </span><span style="color:#5597d6;">head -n </span><span>$</span><span style="color:#5597d6;">maxln
</span><span> </span><span style="color:#5597d6;">success </span><span style="color:#72ab00;">&& </span><span style="color:#b39f04;">exit</span><span> 5 </span><span style="color:#72ab00;">|| </span><span style="color:#b39f04;">exit</span><span> 1;;
</span><span style="color:#72ab00;">esac
</span></code></pre>
<p>As you can see, I simply call <code>dpkg</code> and trim the output. Depending on the
return value, ranger will show the result in a certain way or discard it.</p>
<figure>
<img src="/images/2012-10-17/ranger-deb.png" alt="ranger previewing deb package"/>
<figcaption>ranger previewing a deb package</figcaption>
</figure>
<p>Apart from the added deb previewer, there is another nice detail that one can
easily overlook: The path to this directory, which is actually
<code>/var/cache/apt/archives</code>, is trimmed quite intelligently!</p>
<hr />
<p>In this post, I barely scratched the surface of ranger and there is still a
binders full of possibilities. So, check it out, use it and give those guys a
beer when you see them. Thumbs up for this nice tool!</p>
The Default Shell2012-10-03T00:00:00+00:002012-10-03T00:00:00+00:00
Unknown
https://bloerg.net/posts/the-default-shell/<p>Ever so often, people publish <a href="http://news.ycombinator.com/item?id=4585946">blog posts</a> that tout the <a href="http://zsh.org">Z shell</a>
as the best thing since sliced bread. I also know two guys who use it on a daily
base. Now, as you could imagine, I am a gadgety kind of person, try new things
out and usually “optimize” my workflows but up to now, I was not convinced to
switch entirely. There are two reasons for my reluctance:</p>
<ol>
<li>Bash is <em>the</em> default shell on any Unix-like system I encountered so far.</li>
<li>Most of the zsh features I consider helpful, are already implemented in
a way with the Bash or can be simply extended with little code.</li>
</ol>
<p>I don’t want to convince anyone to switch from zsh to Bash with this post but
just demonstrate some of the things in my <a href="https://github.com/matze/configs/blob/master/.bashrc">.bashrc</a> that could be
considered as zsh builtin alternatives.</p>
<h2 id="completion">Completion</h2>
<p>One of <em>the</em> most cited examples of zsh’s superiority is tab completion. The guy
in the post gives a peek on what’s possible:</p>
<blockquote>
<p>It knows which commands <em>git</em> takes, which hosts are in my hosts file for <em>ssh</em>,
which users my system have when I write <em>chmod</em>, available packages to <em>apt-get</em>,
etc.</p>
</blockquote>
<p>When I read that, I was puzzled, because everything is possible with the
<code>/etc/bash_completion</code> script and auxiliary completion scripts in
<code>/etc/bash_completion.d/</code>. On debian-based systems, this is enabled
out-of-the-box, but in general you would just source the script in your
<code>.bashrc</code> like this: </p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#72ab00;">if </span><span style="color:#b39f04;">[ </span><span style="color:#5597d6;">-f</span><span> /etc/bash_completion </span><span style="color:#b39f04;">] </span><span style="color:#72ab00;">&& ! </span><span style="color:#b39f04;">shopt</span><span> -oq posix</span><span style="color:#72ab00;">; then
</span><span> </span><span style="color:#b39f04;">.</span><span> /etc/bash_completion
</span><span style="color:#72ab00;">fi
</span></code></pre>
<p>What <em>is</em> nice about the zsh completion though, is its navigation-style choice
of different tab-completed options, so instead of <code>cd /usr/share/<tab><tab><tab></code> ad infinitum you can tab once and navigate much
faster between the different completion options.</p>
<h2 id="prompt">Prompt</h2>
<p>From what I can see, zsh prompts tend to be colorful and almost always
overloaded with information. Personally, I just need the absolute minimum:
user name, host (especially when connecting to different machines), and the
current directory. However, I also like to have the path to the current working
directory truncated to maximize the input line. Unfortunately, I have no idea
where this piece of kind originates from, but this is what accomplishes it:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#668f14;">function </span><span style="color:#c23f31;">_prompt_workingdir </span><span>()
</span><span>{
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">pwdmaxlen</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">$(($</span><span style="color:#acb3c2;">COLUMNS</span><span style="color:#72ab00;">/</span><span style="color:#aeb52b;">5</span><span style="color:#d07711;">))
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">trunc_symbol</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"..."
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#b39f04;">[[ </span><span>$</span><span style="color:#5597d6;">PWD </span><span style="color:#72ab00;">== </span><span>$</span><span style="color:#5597d6;">HOME</span><span style="color:#72ab00;">* </span><span style="color:#b39f04;">]]</span><span style="color:#72ab00;">; then
</span><span> </span><span style="color:#5597d6;">newPWD</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"~${</span><span style="color:#acb3c2;">PWD</span><span style="color:#72ab00;">#</span><span style="color:#d07711;">$</span><span style="color:#acb3c2;">HOME</span><span style="color:#d07711;">}"
</span><span> </span><span style="color:#72ab00;">else
</span><span> </span><span style="color:#5597d6;">newPWD</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">${</span><span style="color:#acb3c2;">PWD</span><span style="color:#d07711;">}
</span><span> </span><span style="color:#72ab00;">fi
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#b39f04;">[ </span><span>${</span><span style="color:#72ab00;">#</span><span style="color:#5597d6;">newPWD</span><span>} </span><span style="color:#5597d6;">-gt </span><span>$</span><span style="color:#5597d6;">pwdmaxlen </span><span style="color:#b39f04;">]</span><span style="color:#72ab00;">; then
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">pwdoffset</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">$(( ${</span><span style="color:#72ab00;">#</span><span style="color:#acb3c2;">newPWD</span><span style="color:#d07711;">} </span><span style="color:#72ab00;">- </span><span style="color:#d07711;">$</span><span style="color:#acb3c2;">pwdmaxlen </span><span style="color:#72ab00;">+ </span><span style="color:#aeb52b;">3 </span><span style="color:#d07711;">))
</span><span> </span><span style="color:#5597d6;">newPWD</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"${</span><span style="color:#acb3c2;">trunc_symbol</span><span style="color:#d07711;">}${</span><span style="color:#acb3c2;">newPWD</span><span style="color:#72ab00;">:</span><span style="color:#d07711;">$</span><span style="color:#acb3c2;">pwdoffset</span><span style="color:#72ab00;">:</span><span style="color:#d07711;">$</span><span style="color:#acb3c2;">pwdmaxlen</span><span style="color:#d07711;">}"
</span><span> </span><span style="color:#72ab00;">fi
</span><span> </span><span style="color:#b39f04;">echo </span><span>$</span><span style="color:#5597d6;">newPWD
</span><span>}
</span></code></pre>
<p>zsh (or rather the plugin collection <a href="https://github.com/robbyrussell/oh-my-zsh/wiki/themes">oh-my-zsh</a>?) is also capable of
visualizing Git status information when inside a Git-versioned directory. This
is also possible within Bash using this fine and fast <a href="http://www.opinionatedprogrammer.com/2011/01/colorful-bash-prompt-reflecting-git-status/">snippet</a> from Jo Liss:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#668f14;">function </span><span style="color:#c23f31;">_git_prompt</span><span>()
</span><span>{
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">git_status</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"`</span><span style="color:#acb3c2;">git</span><span style="color:#d07711;"> status</span><span style="color:#acb3c2;"> --porcelain </span><span style="color:#aeb52b;">2</span><span style="color:#72ab00;">>&</span><span style="color:#aeb52b;">1</span><span style="color:#d07711;">`"
</span><span> </span><span style="color:#72ab00;">if ! </span><span style="color:#b39f04;">[[ </span><span style="color:#d07711;">"$</span><span style="color:#acb3c2;">git_status</span><span style="color:#d07711;">" </span><span style="color:#72ab00;">=~ </span><span>Not</span><span style="color:#b3933a;">\ </span><span>a</span><span style="color:#b3933a;">\ </span><span>git</span><span style="color:#b3933a;">\ </span><span>repo </span><span style="color:#b39f04;">]]</span><span style="color:#72ab00;">; then
</span><span> </span><span style="color:#72ab00;">if </span><span style="color:#b39f04;">[ </span><span style="color:#5597d6;">-z </span><span style="color:#d07711;">"$</span><span style="color:#acb3c2;">git_status</span><span style="color:#d07711;">" </span><span style="color:#b39f04;">]</span><span style="color:#72ab00;">; then
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">ansi</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">42
</span><span> </span><span style="color:#72ab00;">elif </span><span style="color:#b39f04;">[[ </span><span style="color:#d07711;">"$</span><span style="color:#acb3c2;">git_status</span><span style="color:#d07711;">" </span><span style="color:#72ab00;">== *</span><span style="color:#d07711;">"??"</span><span style="color:#72ab00;">* </span><span style="color:#b39f04;">]]</span><span style="color:#72ab00;">; then
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">ansi</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">43
</span><span> </span><span style="color:#72ab00;">else
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">ansi</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">45
</span><span> </span><span style="color:#72ab00;">fi
</span><span>
</span><span> </span><span style="color:#668f14;">local </span><span style="color:#5597d6;">branch</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">$(</span><span style="color:#acb3c2;">__git_ps1 </span><span style="color:#d07711;">"%s")
</span><span> </span><span style="color:#5597d6;">test </span><span style="color:#d07711;">"$</span><span style="color:#acb3c2;">branch</span><span style="color:#d07711;">"</span><span> != master </span><span style="color:#72ab00;">|| </span><span style="color:#5597d6;">branch</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">' '
</span><span>
</span><span> </span><span style="color:#b39f04;">echo </span><span style="color:#5597d6;">-n </span><span style="color:#d07711;">'\[\e[0;37;'"$</span><span style="color:#acb3c2;">ansi</span><span style="color:#d07711;">"';1m\]'"$</span><span style="color:#acb3c2;">branch</span><span style="color:#d07711;">"'\[\e[0m\] '
</span><span> </span><span style="color:#72ab00;">fi
</span><span>}
</span></code></pre>
<p>So, I plug these two functions inside my usual prompt like this</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#5597d6;">PS1</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">"`</span><span style="color:#acb3c2;">_git_prompt</span><span style="color:#d07711;">`"'\[\033[1;30m\]me\[\033[0m\]@\[\033[1;30m\]\h\[\033[0m\]:\[\033[0;33m\]$(_prompt_workingdir)\[\033[0m\] '
</span></code></pre>
<p>and achieve this</p>
<figure>
<img src="/images/2012-10-03/prompt.png" alt="Prompt with Git information"/>
</figure>
<p>So, that is my prompt. Context-dependent information and as simple as possible.</p>
<h2 id="navigation">Navigation</h2>
<p>Instead of <em>changing directories</em> with <code>cd</code><sup class="footnote-reference"><a href="#1">1</a></sup> and tab-completion I usually use
the fantastic [autojump][] tool that changes to the directory that matches with
what is supplied as arguments. It needs to learn the paths by going in with <code>cd</code>
but this initial learning phase pays off relatively soon, especially when
changing into deep hierarchies and when confronted with MS-style uppercased
paths. For example,</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ cd /usr/share/gtk-doc/html/gtk3
</span></code></pre>
<p>becomes</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ j u gtk3
</span></code></pre>
<p>Lo and behold!, autojump also supports zsh.</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Okay, I lied. Sometimes, I use it to go into the next child directory. Or
<code>cd -</code> into the previous directory. That’s pretty nifty.
[autojump]: https://github.com/joelthelion/autojump</p>
</div>
<h2 id="command-line-editing">Command-line editing</h2>
<p>In terms of command-line editing both shells provide similar, powerful modes. By
default both use a scheme similar to Emacs, but can also operate with a Vi-like
interface. I only recently began to explore command-line editing and my fingers
are not yet accustomed to it. Anyway, here are some common strokes:</p>
<ul>
<li><code><M-b></code>/<code><M-f></code>: move by word</li>
<li><code><C-b></code>/<code><C-f></code>: move by character</li>
<li><code><C-p></code>/<code><C-n></code>: previous/next historic command line</li>
<li><code><C-a></code>/<code><C-e></code>: go to beginning/end of the line</li>
<li><code><C-u></code>/<code><C-k></code>: delete from current cursor position to the beginning/end</li>
</ul>
<p>You should know them not only because they are easier to type than the arrow and
page keys, but because they are available to any application that offers a
command-line interface based on <a href="http://www.gnu.org/software/readline/">readline</a>. For example, to rename a tmux window I
am much faster with <code><C-a>,<C-u></code> than using the backspace key.</p>
<hr />
<p>So, this is basically my Bash setup with some simple yet powerful enhancements
to the stock experience. I am pretty satisfied with this setup but if you have
anything that might convince me to switch to zsh, don’t hesitate to tell me.</p>
TicGit Issue Tracker2012-09-24T00:00:00+00:002012-09-24T00:00:00+00:00
Unknown
https://bloerg.net/posts/ticgit-issue-tracker/<p><a href="https://github.com/jeffWelling/ticgit">TicGit</a>-ng is an <a href="http://en.wikipedia.org/wiki/Issue_tracking_system">issue tracker</a> and – as the name suggests – tightly
coupled within a Git repository. Thus, the idea is to keep the issue tracking
system along with the source rather than setting up a web server instance with
one of the many open systems or paying for an equivalent service.</p>
<p>Since Lucid, TicGit is packaged for Ubuntu, so you can simply grab the CLI tool
and a simple web server with</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo apt-get install ticgit ticgitweb
</span></code></pre>
<p>Alternatively, you can also install the <code>gem</code> tool and pull everything from the
Rubygems site</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo apt-get install ruby rubygems
</span><span>sudo gem install git sinatra haml sass
</span></code></pre>
<p>Before adding tickets, you have to run <code>ti init</code> in a directory that contains
a Git repository. <code>ti new</code> fires up <code>$EDITOR</code>, to let you enter a ticket summary
and additional tags. You can also take a short cut and enter a new ticket with
<code>ti new -t 'summary'</code>. <code>ti list</code> lists all <em>open</em> tickets.</p>
<p>Now, if you want to use one of the more advanced commands such as <code>comment</code>,
<code>state</code> etc., you have to <code>checkout</code> a ticket by its <em>TicId</em> or its position on
the list. To me this is a little bit awkward but okay. Although you could simply
push and pull the TicGit branch to your remote, it is probably a better idea to
use the builtin <code>sync</code> command.</p>
<p>Because it is quite cumbersome to remember and type all these commands, <a href="http://www.cmdln.org/2008/04/11/command-line-ticketing-system/">Nick
Anderson</a> wrote a Bash completion script that completes the options. I
enhanced the script with ticket completion and some more commands that made it
into TicGit since 2008. To enable Bash completion, put the <a href="http://gist.github.com/3777344">script</a> into
<code>/etc/bash_completion.d/</code>.</p>
<p>To start the TicGit web server, simply change into a TicGit-enabled repository
and run <code>ticgitweb</code>. Point your browser to <code>0.0.0.0:4567</code> and you’ll be greeted
with a simple interface to TicGit:</p>
<figure>
<img src="/images/2012-09-24/ticgit.png" alt="Ticket details"/>
<figcaption>Details of a ticket.</figcaption>
</figure>
<p>However, as far as I can see there are also some drawbacks to the TicGit
approach:</p>
<ul>
<li>Although – being based on Git – it allows for a very nice distributed issue
tracking system, I cannot imagine it to be used by a larger community as it
simply does not scale: Everyone would have to pull from everyone to have a
consistent view of all issues.</li>
<li>In the same vain, TicGit is probably a little less appealing to people not
friends with the command line.</li>
<li>Performance is somewhat lacking. Most commands are executed after a noticeable
latency. This may or may not be caused by Ruby. Who knows?</li>
<li>Extensibility is non-existent. For example, there are four pre-defined ticket
states (open, hold, invalid, resolved) but you cannot change them nor add new
ones. Also mandatory features found in larger tracking systems such as
versions and milestones are missing.</li>
</ul>
<p>Despite these points, , TicGit is a perfect solution to keep track of bugs and
issues for smaller projects such as my blog here.</p>
Taking notes with Markdown and Pandoc2012-09-14T00:00:00+00:002012-09-14T00:00:00+00:00
Unknown
https://bloerg.net/posts/taking-notes-with-markdown-and-pandoc/<p>For the last couple of years, taking notes on a variety of subjects was driving
me nuts: I tried it on single sheets of paper (almost no organization and easily
lost), within a paper notebook<sup class="footnote-reference"><a href="#1">1</a></sup> (hard to edit once a page is full) and I
tried a combination of web-based tools such as delicious, Asana, Evernote and
several others I cannot remember anymore. The drawback of the latter is
apparent: once you are disconnected you are out of luck.</p>
<p>Due to this mess, I began to write down new notes in the format to rule them
all: Markdown; one file covering a single subject. Although I could serve and
edit the Markdown files with <a href="/2012/08/08/node-wiki-and-my-first-nodejs-seps.html">Nodewiki</a>, I decided to generate a single,
static HTML page with Pandoc to review my notes.</p>
<h2 id="rendering-markdown-content-with-pandoc">Rendering Markdown content with Pandoc</h2>
<p><a href="http://johnmacfarlane.net/pandoc/">Pandoc</a> is a document conversion tool that accepts “markdown,
reStructuredText, textile, HTML, DocBook, or LaTeX” and outputs amongst others
HTML (normal pages and slide shows), EPUB and the mentioned markup formats. </p>
<p>Multiple input files can be passed to Pandoc, thus generating a single HTML
file is as simple as</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>pandoc -s *.md > foo.html
</span></code></pre>
<p>Because the default Pandoc HTML output left everything to be desired, I
<a href="https://github.com/matze/pandoc-notes">wrote</a> an HTML template with a toggable table of contents, a style sheet
that is easy on the eyes and a Makefile that takes care of generating the final
HTML page and pushing it to my server.</p>
<p><em>UPDATE</em>: I abandonded that repository for a slightly more generic
<a href="https://gist.github.com/4194445">approach</a> that I describe <a href="/2012/12/05/pandoc-and-tex.html">here</a>.</p>
<p>Initially, I also tried <a href="http://www.unexpected-vortices.com/sw/gouda/#simple-template">Gouda</a> to generate one HTML page per Markdown file.
But I didn’t like the rigid structure of the final result and hence sticked with
this setup. However, should something more suitable arrive on the scene, I will
be prepared with my plain text Markdown notes.</p>
<h2 id="synchronization-with-bitpocket">Synchronization with Bitpocket</h2>
<p>Instead of using Git, I store the notes in a plain directory and synchronize it
with <a href="https://github.com/sickill/bitpocket">Bitpocket</a> across all machines:</p>
<blockquote>
<p>In short, Bitpocket is a small but smart bash script that does two-way
directory synchronization resembling Dropbox sync. Simply it just uses rsync
to make the actual sync. It runs rsync twice: first syncing from remote to
local machine, then from local to remote machine. This way all new files that
appeared on remote are fetched to local machine and all new locally created
files are replicated on remote machine.</p>
<footer>— <a href="http://ku1ik.com/2011/07/18/bitpocket-as-a-dropbox-alternative.html">Bitpocket as a Dropbox alternative</a></footer>
</blockquote>
<p>Although, I lose the ability to go back to a previous state as with Git, I don’t
have to do the “add, write comment, commit and push” dance, to update a simple
note. Until now, I haven’t yet had any problems with Bitpocket in terms of
stability or lost data. Knockin’ on wood …</p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>I still use a paper notebook for taking notes during verbal meetings or on my way without a PC, though. Long term information is transfered into my Markdown-based system later on.</p>
</div>
Raspberry Pi Audio Update2012-09-03T00:00:00+00:002012-09-03T00:00:00+00:00
Unknown
https://bloerg.net/posts/raspberry-pi-audio-update/<p>As promised, this post is the final update on my journey of building
high-quality audio solution based on the Raspberry Pi. As I said in my
[last post][], I ordered the Terratec Aureon Dual USB sound “card” (around 16
EURs) and a TOSLINK cable. The card arrived in a timely manner and, boy!, this
is what I’d call plug ‘n’ play. After connecting it with a USB extension cord to
the board<sup class="footnote-reference"><a href="#1">1</a></sup> the kernel recognized it immediately:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ lsusb
</span><span>Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
</span><span>Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
</span><span>Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
</span><span>Bus 001 Device 004: ID 0586:341e ZyXEL Communications Corp. NWD2105 802.11bgn Wireless Adapter [Ralink RT3070]
</span><span>Bus 001 Device 005: ID 0ccd:0077 TerraTec Electronic GmbH Aureon Dual USB
</span></code></pre>
<p>To my surprise, Poettering’s [brain child][pulse-audio] has not yet arrived in
the Raspian distribution, thus sound output is still managed by ALSA. Of
course, when I tried listening with [mpd][], music was only output on the crappy
on-board jack. So I wasted a lot of time with adapting <code>/etc/asound.conf</code> and
whatnot until I realized that I could simply specify a different output and
mixer device in <code>/etc/mpd.conf</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>audio_output {
</span><span> type "alsa"
</span><span> name "My ALSA Device"
</span><span> device "hw:1,0" # use the external USB sound card
</span><span> format "44100:16:2"
</span><span> mixer_device "hw:1" # same here
</span><span> mixer_control "Speaker" # the mixer for the analog jack
</span><span> mixer_index "0"
</span><span>}
</span></code></pre>
<p>Now, mpd outputs on the USB sound card and analog volume can be controlled with
any mpd client. Compared to the on-board DAC, the Terratec is huge step forward;
no hissing, pops and noises at high frequencies. So this is it: <em>my audio setup
is complete.</em></p>
<div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup>
<p>Even though the Zyxel adapter is rather small, both devices do not fit in the dual USB socket.
[last post]: /2012/08/26/raspberry-pi-with-wifi-and-audio.html
[pulse-audio]: http://pulseaudio.org
[mpd]: http://mpd.wikia.com/wiki/Music_Player_Daemon_Wiki</p>
</div>
A Jekyll ditaa Plugin2012-08-30T00:00:00+00:002012-08-30T00:00:00+00:00
Unknown
https://bloerg.net/posts/jekyll-ditaa-plugin/<p>In my <a href="/2012/08/26/raspberry-pi-with-wifi-and-audio.html">last post</a>, I used a little diagram to describe my audio setup. Some
simple boxes and arrows, just like this:</p>
<p>{% ditaa %}
+—–+ /—–<br />
| Foo |——>| Bar |
+—–+ -––/
^
/—–+ |
| Baz |—=—+
+—–+
{% endditaa %}</p>
<p>However, this would not be in the spirit of hacking if I prepared this with a
graphics program. Instead I used <a href="http://ditaa.sourceforge.net/">ditaa</a> to transform this ASCII text</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>+-----+ /-----\
</span><span>| Foo |------>| Bar |
</span><span>+-----+ \-----/
</span><span> ^
</span><span> /-----+ |
</span><span> | Baz |---=---+
</span><span> +-----+
</span></code></pre>
<p>into the image above. This is working just fine but it’s annoying to</p>
<ol>
<li>write the markup,</li>
<li>process it with ditaa and</li>
<li>insert a link of the generated image in the markdown document.</li>
</ol>
<p>Sure, it would be much more straightforward to just type the ditaa markup inline
into the markdown document, wouldn’t it? Because I couldn’t find a plugin that
implements this behaviour, I wrote it <a href="https://github.com/matze/jekyll-ditaa">myself</a>. The plugin registers the
<code>ditaa</code> block with Liquid, tries to generate an image and place the image link
right into the final HTML post:</p>
<p>{% raw %}
# My neat little boxes</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>{% ditaa %}
</span><span>+-----+ +-----+
</span><span>| Foo |----->| Bar |
</span><span>+-----+ +-----+
</span><span>{% endditaa %}
</span><span>
</span><span>This is how they roll!
</span></code></pre>
<p>{% endraw %}</p>
<p>In case, ditaa is not installed or the image could not be generated, the plugin
falls back to output the plain ASCII diagram in a <code><pre></code> block.</p>
<h2 id="limitations">Limitations</h2>
<p>With <a href="http://liquidmarkup.org/">Liquid</a> – the templating engine of Jekyll – there is only one way to
implement this feature and that is sub-classing <code>Liquid::Block</code>. Unfortunately,
a block is just supposed to generate HTML from whatever is in inside but not new
assets in the file system. For generating new content, the <code>Jekyll::Generator</code>
should be used but this is only working on a higher, per-site level.</p>
<p>Now, I generate an image for each ditaa block and store that in an <code>images/</code>
directory visible to Jekyll so that Jekyll copies it whenever it re-generates
the site. So in a way, Jekyll does not know that it created that image itself.
This becomes noticeable when the site is deployed without auto-regeneration
(e.g. when pushing to a Git repository and generating a fresh site). In this
case, a second Jekyll pass is necessary. Not cool.</p>
<h2 id="ps-the-ribbons">PS: The ribbons</h2>
<p>You probably have noticed the “Fork me on GitHub” ribbon. I always liked that
these visual cues help people immediately recognize if there is something to
check out and fork off. From now on, you will see this ribbon on all pages that
feature a project that I put on GitHub, along with the <a href="http://octodex.github.com/">Octocats</a>
<i class="icon-github"> </i>on the landing page.</p>
<p>I used a little Jekyll magic, to avoid putting this in manually for each post
that refers to a repository: Whenever I want to link to one of my repositories,
I add a <code>repo</code> URL variable in the <a href="https://github.com/mojombo/jekyll/wiki/YAML-Front-Matter">YAML front matter</a>. The Liquid
template for a blog post checks if this variable exists and inserts the image
template with the <code>repo</code> URL. Simple but powerful.</p>
Wireless Raspberry Pi and Audio2012-08-26T00:00:00+00:002012-08-26T00:00:00+00:00
Unknown
https://bloerg.net/posts/raspberry-pi-with-wifi-and-audio/<h2 id="wireless-lan">Wireless LAN</h2>
<p>Setting up wireless LAN on a Raspberry Pi is straightforward as long as you get
a reasonably supported adapter. I bought a Zyxel NWD2105 for under ten euros
which is working out of the box with the drivers delivered with Raspbian.</p>
<figure>
<img src="/images/2012-08-26/zyxel.jpg" />
</figure>
<p>I wanted to access the RPi with a static IP address, so I added an
entry for the wireless interface to <code>/etc/network/interfaces</code></p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>auto wlan0
</span><span>iface wlan0 inet static
</span><span>address 192.168.2.124
</span><span>gateway 192.168.2.1
</span><span>netmask 255.255.255.0
</span><span>wpa-conf /etc/wpa.conf
</span></code></pre>
<p>and put in security credentials in <code>/etc/wpa.conf</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>network={
</span><span> ssid="CaseSensitive Router SSID"
</span><span> proto=RSN
</span><span> key_mgmt=WPA-PSK
</span><span> pairwise=CCMP TKIP
</span><span> group=CCMP TKIP
</span><span> psk="password"
</span><span>}
</span></code></pre>
<p>Now, there was one issue: Wireless is only working when an ethernet cable is
connected. I am not joking! I could see both interfaces with <code>ifconfig</code> and
verify that the WLAN adapter was connected with <code>iwconfig</code>. But when I unplugged
the ethernet cable and rebooted the RPi, I could not connect anymore. Funnily,
the RPi itself talked to some services (<code>avahi-browse -a</code> showed some activity)
but I could not talk to the RPi. A question on <a href="http://unix.stackexchange.com/questions/42734/networking-is-not-starting-up">Unix@SE</a> proved, that I am
not the only person seeing the interfaces go crazy. One reply suggested to
change <code>auto eth0</code> to <code>allow-hotplug eth0</code> but this was not working for me. So,
again I had to pull out the hammer and just remove the whole <code>eth0</code> entry.</p>
<h2 id="streaming-music-to-my-stereo-system">Streaming music to my stereo system</h2>
<p>In an <a href="/2012/08/05/music-sharing-pain-on-a-raspbery-pi.html">earlier post</a>, I described my half-assed solution to stream
from the RPi to my workstation. This is working fine and the next step is to
serve the music to my Kenwood stereo system. So installed the <a href="http://mpd.wikia.com/wiki/Music_Player_Daemon_Wiki">music player
daemon</a> and mpc on the RPi and <a href="http://ncmpcpp.rybczak.net/">ncmpcpp</a> on my local desktop. After
starting the daemon, it complained that it could bind to <code>localhost</code> but not to
<code>[:1]</code>, whatever that means. Nothing really helped except changing the
<code>bind_address</code> from <code>localhost</code> to the network interfaces IP address.</p>
<p>I also installed <a href="https://play.google.com/store/apps/details?id=com.namelessdev.mpdroid">mpdroid</a>, an MPD client for my Nexus S Android smart phone.
Although it could not connect to a Zeroconf address – which is
<code>raspberrypi.local</code> –, I could connect just fine to the static IP address.</p>
<p>Now, my whole digital audio setup looks like this:</p>
<p>{% ditaa -E %}
/––+ DAAP /—–+—–+ Audio /––––+
| PC |<——| RPi | MPD |—––>| Stereo |
+––+ +—–+—–+ +––––+
| ^ ^
| ncmpcpp | | mpdroid /———+
+––––=––––+ +––=––| Nexus S |
+———+
{% endditaa %}</p>
<p>I can listen to digital music without turning my workstation on or spending
money on an expensive third-party solution like the Logitech Squeezebox or the
Sonos streaming system.</p>
<p>Unfortunately, there is one big flaw in my setup: I use the 3.5mm analog jack
for streaming audio to my stereo. But the quality of this output is unbearable.
There is a constant noise and higher frequencies are accompanied by strange
popping sounds. Someone at the official forums took the time to analyse the
problem:</p>
<blockquote>
<p>I’ve done some quick tests on the analogue audio quality. I used audacity to
generate a sine wave sweep from 20Hz to 20kHz and saved that as a .wav to play
on the Pi. I then played one channel through my Hifi and ran the other channel
to an oscilloscope. The signal looks and sounds clean up to about 800Hz, then
the distortion start to creep in, by a couple of kHz you can see obvious steps
on the edges of the waveform. From then up it looks more and more digital,
basically less and less bits, at about 10kHz it’s essentially a square wave.
Presumably this is caused by a relatively low PWM frequency and the minimal
filtering, according to the schematic it’s just a single RC low pass filter.</p>
<footer>— <a href="http://www.raspberrypi.org/phpBB3/viewtopic.php?p=103330#103170">slacker</a></footer>
</blockquote>
<p>One of the official developers wrote:</p>
<blockquote>
<p>We are limited to 100MHz, so we can get to 11 bits @ 48kHz.</p>
<p>We did play with 6 bit samples, and an oversampling factor of 8 using a sigma
delta algorithm, but it sounded worse.</p>
<footer>— <a href="http://www.raspberrypi.org/phpBB3/viewtopic.php?p=103330#103330">dom</a></footer>
</blockquote>
<p><em>Argh!</em></p>
<p>Now, after ditching the analog output, I have two options. First, I could use
the integrated HDMI output. Unfortunately, my stereo has only a digital TOSLINK
input, no HDMI and I don’t want to buy one of those little boxes that can mix
and match TOSLINK and HDMI. They are a too expensive and overpowered for my
problem. The other option, that I am realizing now, is to buy a USB sound card
that features a TOSLINK output. A small, decently priced sound card is the
TerraTec AureonDualUSB, that I just ordered from Amazon. How this turns out is
to be told about in another episode of “How I met my perfect audio solution”.</p>
Updates in Vim Land2012-08-23T00:00:00+00:002012-08-23T00:00:00+00:00
Unknown
https://bloerg.net/posts/updates-in-vim-land/<p>In this post, I’ll show you the latest additions to my collection of useful Vim
plugins: The <em>Vundle</em> plugin manager and the <em>dwm.vim</em> window manager not to be
confused with the <em>dwm</em> window manager.</p>
<h2 id="vundle-vs-pathogen">Vundle vs. Pathogen</h2>
<h3 id="status-quo-ante">Status quo ante</h3>
<p>Just like many of you, I keep my “dot files” version controlled and backed up at
<a href="https://github.com/matze/configs">GitHub</a>. Moreover, I used <a href="https://github.com/tpope/vim-pathogen">Pathogen</a> to bundle up all Vim
plugins instead of having plugin files scattered around in <code>autoload/</code>,
<code>plugin/</code>, etc. To automatize the procedure of setting up my configuration
on a new machine, I came up with a mix of scripts:</p>
<ul>
<li>A shell script that creates symlinks from the repositories dot files.</li>
<li>A Python script that checked out Git repositories of Vim plugins from a list
of GitHub URLs and curl’d my favorite color scheme.</li>
<li>A Makefile with two targets: <code>config</code> that called the shell script and <code>vim</code>
that called the Python script. Yes, it was that pointless.</li>
</ul>
<p>This approach worked just fine for me but is obviously an ugly crutch. Moreover,
I constantly forgot the <code>make</code> targets …</p>
<h3 id="vundle">Vundle</h3>
<p><a href="https://github.com/gmarik/vundle">Vundle</a> is a drop-in replacement for Pathogen which means all plugins living
under <code>bundle/</code> are managed in the same way as with Pathogen. Unfortunately,
Pathogen does not know how to get the plugins, so it depends on some external
magic to find them under <code>bundle/</code>. That is the reason why I have used this
awful Python script and other people import plugin repositories via <code>git submodule</code> into their configuration repositories. Vundle solves this problem
because it knows how to fetch plugins from different sources.</p>
<p>All you have to add to your <code>.vimrc</code> are two lines of boilerplate and a list of
plugins from GitHub, VimScript and non-GitHub Git repositories. This smartness
is a shining example of <em>convention over configuration</em>:</p>
<pre data-lang="vim" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-vim "><code class="language-vim" data-lang="vim"><span style="color:#b39f04;">filetype</span><span> off
</span><span>
</span><span style="color:#b39f04;">set</span><span> rtp+</span><span style="color:#668f14;">=~</span><span style="color:#c49a39;">/.vim/</span><span>bundle/vundle
</span><span>call </span><span style="color:#b39f04;">vundle#rc</span><span>()
</span><span>
</span><span>Bundle </span><span style="color:#d07711;">'mileszs/ack.vim'
</span><span>Bundle </span><span style="color:#d07711;">'Raimondi/delimitMate'
</span><span>Bundle </span><span style="color:#d07711;">'Shougo/neocomplcache'
</span><span>Bundle </span><span style="color:#d07711;">'Shougo/neocomplcache-snippets-complete'
</span><span>Bundle </span><span style="color:#d07711;">'Lokaltog/vim-powerline'
</span><span>Bundle </span><span style="color:#d07711;">'kien/ctrlp.vim'
</span><span>Bundle </span><span style="color:#d07711;">'tpope/vim-commentary'
</span><span>Bundle </span><span style="color:#d07711;">'tpope/vim-markdown'
</span><span>Bundle </span><span style="color:#d07711;">'tpope/vim-fugitive'
</span><span>Bundle </span><span style="color:#d07711;">'nvie/vim-flake8'
</span><span>Bundle </span><span style="color:#d07711;">'matze/dwm.vim'
</span><span>
</span><span style="color:#b39f04;">filetype</span><span> on
</span></code></pre>
<p>So, instead of having the plugins magically appear, I can tell <em>from</em> my
<code>.vimrc</code> what plugins I want to fetch and where from. To install the plugins,
you simply call <code>:BundleInstall</code> in a Vim session and watch Vundle installing
them:</p>
<figure>
<img src="/images/2012-08-23/vundle-installing.png" />
</figure>
<p>Plugins are updated with the <code>:BundleInstall!</code> (note the bang) and deinstalled
by removing the <code>Bundle</code> line from the <code>.vimrc</code> and calling <code>:BundleClean</code>.
Very simple.</p>
<h3 id="status-quo">Status quo</h3>
<p>Now, instead of maintaining a Makefile, a shell script and the Python updater I
distilled everything in a single <a href="https://github.com/matze/configs/blob/master/setup.sh">setup.sh</a> that just updates the symlinks
clones Vundle. The rest is taken care of by Vundle itself. Life couldn’t be
easier.</p>
<h2 id="dwm-vim-vim-windows-done-right">dwm.vim – Vim windows done right</h2>
<h3 id="vim-windows">Vim windows</h3>
<p><a href="http://vimdoc.sourceforge.net/htmldoc/windows.html">Windows</a> are part of Vim for a very long time and probably
confusing first time users since its very beginning. Other than windows being
<em>views</em> on <em>buffers</em> there is no direct relation between them. There can be
windows that don’t show any buffer and there can be multiple windows that show
the same buffer but in a different way. That’s the power of flexibility coming
at the cost of complexity.</p>
<p>For most of the time, I never really bothered with windows because I was mostly
switching between buffers in a single window with the <a href="https://github.com/fholgado/minibufexpl.vim">minibufexplorer</a>
plugin. So, why bothering with windows then? Because I ditched minibufexplorer
some time ago and because it is often a good idea to see the contents of
different buffers at the same time. For this reason, I had key maps to create
windows and move between them. As you can imagine, the layout was static and
only determined by the order I opened them. Not very flexible and thus not very
often used by myself. Fortunately, there is a new and surprisingly simple plugin
on the Vim landscape that solves the problem of rigid window layouts.</p>
<h3 id="dwm-is-not-dwm-but-almost">dwm is not dwm but almost</h3>
<p><a href="https://github.com/spolu/dwm.vim">dwm.vim</a> is a plugin named after <a href="http://dwm.suckless.org/">dwm</a>, a <em>tiling</em> window manager and
origin of my window manager of choice, <a href="http://awesome.naquadah.org/">awesome wm</a>. According to what I read
(I never used dwm), dwm.vim’s mechanics are pretty similar to dwm’s that sports
a single window pane M and one side and a stacked window pane S on the other.</p>
<p>Once installed – preferably with Vundle – you use <code><C-N></code> to create a new
window with an empty buffer in pane M. All other windows are pushed to pane S
and stacked atop. You switch to the next (clockwise) window with <code><C-J></code> and to
the previous with <code><C-K></code>. A stacked window is focused and moved to pane M with
<code><C-H></code>. In my setup, I use <code><C-X></code> to close the current window. This was
originally set to <code><C-C></code> by the author of the plugin. However, because I use
<code><C-C></code> to go from insert mode to normal mode, I sometimes accidentally closed
the window without intending to do so. This graphic sums the mechanics up pretty
well, I guess:</p>
<figure>
<img src="/images/2012-08-23/dwm.vim.png" />
</figure>
<p>Although I just started using dwm.vim, I already feel the same potential gains I
experienced after installing <a href="https://github.com/kien/ctrlp.vim">CtrlP.vim</a>. So, thumbs up (or like or +1), for
this nice little plugin.</p>
<p><em>Update</em>: I contributed a patch that stops dwm.vim from mapping all those keys
and makes it much easier to override <code><C-C></code> with <code><C-X></code>.</p>
Redirecting bloerg.net/+ to my Google+ profile2012-08-21T00:00:00+00:002012-08-21T00:00:00+00:00
Unknown
https://bloerg.net/posts/bloerg-net-plus/<p>Have you seen it? The link in the top bar with the Google+ logo? It used to be</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>https://plus.google.com/113162113247238651811/
</span></code></pre>
<p>an unwieldy, hard-to-remember URL. For this reason, people use services such as
<a href="http://gplus.to">gplus.to</a> where you can define an URL pointing to a Google+
profile. However, I consider this a bad practice because the
service is not affiliated with Google and prone to abuse.</p>
<p>I wasn’t really aware that is a common practice to append a <code>+</code> to a URL and
redirecting to the profile until I read <a href="http://blog.stefan-betz.net/2012/08/20/sch%C3%B6ne-google+-urls-f%C3%BCr-die-eigene-domain/">this</a> blog post by Stefan Betz.
I didn’t even notice that Google <a href="http://google.com/+">themselves</a> use this
approach to save some characters. Looks like I’m getting old and ignorant.</p>
<p>If you serve your site with nginx, all you have to do is to add the following
redirect in <code>/etc/nginx/nginx.conf</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>location / {
</span><span> # ...
</span><span> rewrite ^(/\+)$ https://plus.google.com/113162113247238651811/ permanent;
</span><span>}
</span></code></pre>
<p>This is a nice and simple alternative to <a href="http://en.wikipedia.org/wiki/Vanity_URL">vanity URLs</a> that are
still reserved for <a href="https://plus.google.com/+britneyspears">celebrities</a> and <a href="https://plus.google.com/+toyota">companies</a>.</p>
<p><em>Update</em>: The Google+ link in the top bar points once again to the old, unwieldy
URL. Google dislikes redirects when using the <code>rel="author"</code> attribute that is
used to link a website with a “corresponding” Google+ page.</p>
Nodewiki and My First Node.js Steps2012-08-08T00:00:00+00:002012-08-08T00:00:00+00:00
Unknown
https://bloerg.net/posts/node-wiki-and-my-first-nodejs-seps/<p>How do you get me interested? Mention the words Git, Markdown and Vim. Okay,
okay, Vim does not play a primary role in this post about the awesome
<a href="https://github.com/nhoss2/nodewiki/">Nodewiki</a> that is developed by <a href="http://labs.im">poor
guy</a> <em>nhoss2</em>. Nodewiki is a server software that displays
formatted Markdown files in your browser. You can edit the files inside the
browser and record the changes with Git. This is a perfect company for my
technical notes which I already keep as Markdown files. </p>
<figure>
<img src="/images/2012-08-08/2012-08-08-nodewiki.jpg" />
</figure>
<h2 id="installation">Installation</h2>
<p>Nodewiki is based on <a href="http://nodejs.org">Node.js</a>, a framework that let’s you
write web servers in JavaScript and run them in Chrome’s V8 Javascript engine.
The installation is incredibly easy with an up-to-date Ubuntu:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>sudo apt-get install nodejs nodejs-dev npm
</span><span>sudo npm install nodewiki -g
</span></code></pre>
<p>Simply change into a directory with your Markdown files, type</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>nodewiki
</span></code></pre>
<p>to start the server and point your browser to <code>127.0.0.1:8888</code> and you are ready
to view and edit your files. If you want to use version control with Git, you
need to initialize the Git repository, add your files and start the server with
the <code>--git</code> flag:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#b39f04;">cd</span><span> notes/
</span><span style="color:#5597d6;">git</span><span> init
</span><span style="color:#5597d6;">git</span><span> add .
</span><span style="color:#5597d6;">git</span><span> commit</span><span style="color:#5597d6;"> -m </span><span style="color:#d07711;">"initial commit"
</span><span style="color:#5597d6;">nodewiki --git
</span></code></pre>
<p>However, I didn’t find this feature particularly useful for two reasons: 1. I
already sync my notes with <a href="https://github.com/sickill/bitpocket/">bitpocket</a>
and I don’t want to annotate all the changes with commit messages when I’m <em>not</em>
editing the files with Nodewiki. 2. The autogenerated commit messages are not
very useful.</p>
<p>I also wanted to tinker a bit with the code, to see how things work in the
Javascript world. But being never exposed to the Node.js ecosphere, I had a hard
time to get this stuff running. Although I could run the globally installed
Nodewiki software, Node always complained about missing dependencies when
running from the working tree. The solution is astoundingly simple: Node always
looks for modules in a <code>node_modules/</code> directory in the same path where the
<code>$app.js</code> script is located. This means I just had to install all dependencies
again in the local working tree:</p>
<pre data-lang="bash" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#b39f04;">cd</span><span> nodewiki
</span><span style="color:#5597d6;">npm</span><span> install
</span></code></pre>
<p>One the one hand this solves the problem of having incompatible versions of one
package needed by two different apps, but on the other hand <code>npm</code> will most
likely dump lots of duplicate packages on my machine.</p>
<h2 id="final-thoughts">Final Thoughts</h2>
<p>Although the software looks very good already, one important point is missing:
inter-page linking. Really. I needed to consult the <code>Todo.md</code> to be sure that this
is not yet implemented. Another question that I still need to answer for
myself is, if I should put this on a server. It has a certain charme to be able
to edit notes wherever there is a browser available. But I certainly also want to
edit them with Vim from the command line.</p>
Music Sharing Pain on a Raspberry Pi2012-08-05T00:00:00+00:002012-08-05T00:00:00+00:00
Unknown
https://bloerg.net/posts/music-sharing-pain-on-a-raspbery-pi/<p>Like every cool kid on the ’net, I now own a <a href="http://raspberrypi.org">Raspberry
Pi</a>. At first, I wasn’t really sure what to do with the
RPi but it finally dawned on me, to use it as a music streaming server. With a
mere 3.5 W for the whole board, it is reasonable to keep it powered on, even
when considering my wallet. Moreover, I planned to attach it to my little
<em>Kenwood K-521</em> stereo system and use the RPi as another source of music with
MPD.</p>
<figure>
<img src="/images/2012-08-05/2012-08-05-rpi.jpg" />
</figure>
<p>For starters, I installed the officially supported
<a href="http://raspbian.org">Raspbian</a> distribution on a SD card that I had lying
around. Unfortunately, I wrote the image with </p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>dd bs=1M if=2012-07-15-wheezy-raspbian.img of=/dev/sdc1
</span></code></pre>
<p>which was not working. Of course, <code>dd</code> needs to write to the device <code>/dev/sdc</code>
not to the partition <code>/dev/sdc1</code>. After fixing this minor problem, things were
alright: the LEDs are blinking and I could connect via SSH. Until I started
setting up music streaming …</p>
<p>There is an
<a href="http://en.wikipedia.org/wiki/List_of_streaming_media_systems">endless</a> list of
streaming server solutions. However, in my case I want to select a particular
title on the RPi that in turn streams the music to my local machine. For this
purpose, there are two general solutions:
<a href="http://en.wikipedia.org/wiki/Digital_Audio_Access_Protocol">DAAP</a> developed by
Apple for its iTunes platform (well, okay …) and
<a href="http://en.wikipedia.org/wiki/Universal_Plug_and_Play">UPnP</a>/<a href="http://en.wikipedia.org/wiki/Dlna">DLNA</a>
developed by a consortium and often used in settop appliances. Of course I need
server software on the RPi and a suitable client on my local machine and this is
where the dilemma begins.</p>
<p>I am not overly fond of a proprietary protocol developed by Apple, so I tried
DLNA first. In the Debian packages there are packages for
<a href="http://sourceforge.net/projects/minidlna/">minidlna</a> and
<a href="https://live.gnome.org/Rygel/">Rygel</a>, two servers that fit the bill. Back on
my Ubuntu 12.04 desktop things aren’t that nice anymore. I wanted to use
Rhythmbox or Banshee but both lost native UPnP client capabilities some time
ago. UPnP support comes back with the Grilo framework, but this is currently not
enabled on Ubuntu builds. I tried to built Rhythmbox from source but that failed
somewhere in the build process. VLC is capable of UPnP but sucks as a music
player the same way that XBMC is way too big for my purposes.</p>
<p>So, back to the drawing board. After browsing Rhythmbox’ and Banshee’s feature
list, I knew both players receive content streamed with DAAP. Unfortunately,
there are virtually no working DAAP servers on Debian ARM based Raspbian
distribution. In the old days, one would install the <code>mt-daapd</code> server that has
a questionable code quality. For better or worse, there are no <code>mt-daapd</code>
packages for Raspbian. Then I read about the
<a href="http://www.flyn.org/projects/dmapd/index.html">dmapd</a> DAAP server which has to
be built from source. But again, no luck. Then I learnt, that a developer took
the old <code>mt-daapd</code> code, rewrote it in a sane way and called it
<a href="https://github.com/jasonmc/forked-daapd">forked-daapd</a>, which is the de facto
DAAP server nowadays. Well, on i386 and x86_64 at least. For some reason, he
started developing a <code>gcd</code> branch that requires the Clang compiler which does
not yet support ARM architectures. Because the Debian maintainers do not want to
ship a <code>forked-daapd</code> that does not receive security updates in the future, they
removed the non-Clang version from the repository. In despair, I took the latest
<code>forked-daapd</code> release that compiles with <code>gcc</code> and tried to build it. Turns
out, that in his old code base he relies on <code>libevent</code> 1.4 whereas the Debian
maintainers decided to ship <code>libevent</code> 2.0 …</p>
<p>What to do? The simple answer: pulling in Mono and installing the Tangerine
DAAP “server”. It works pretty well, but because it is not designed to be a real
server software, there are no init scripts and it does not integrate with
common system software. To get a running server after boot, I pulled out the
hammer and added this to <code>/etc/rc.local</code>:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>/usr/bin/tangerine & disown
</span></code></pre>
<p>This is horrible in so many ways, but after wasting so much time, I was finally
looking for a fast rather than nice way. Although Tangerine is written in C#, it
is remarkably light weight:</p>
<figure>
<img src="/images/2012-08-05/2012-08-05-load.png" />
</figure>
<p>This is just what I wanted:</p>
<figure>
<img src="/images/2012-08-05/2012-08-05-banshee.png" />
</figure>
<p>This solution is anything but perfect and still misses some important
components: I will have to buy a small WiFi USB adapter, a real power supply (I
am currently using the charger for my <em>Nexus S</em>) and an external HDD case where
I will put on the music. Then I have to hook it up to the Kenwood system and
configure MPD. Hopefully, the digital-to-analog converter is not as bad as
people are complaining on the Raspberry Pi forums. Finally, I have to think
about a real case for the Raspberry Pi. I hope these steps will not be as
annoying as installing a media streaming server.</p>
Making-of bloerg.net2012-07-28T00:00:00+00:002012-07-28T00:00:00+00:00
Unknown
https://bloerg.net/posts/making-of-bloerg-net/<p>Recently, I was approached and asked how this blog was made. Well, the
foundation is <a href="https://github.com/mojombo/jekyll/">Jekyll</a>, described by
its developers as </p>
<blockquote>
<p>a simple, blog aware, static site generator. It takes a template directory
(representing the raw form of a website), runs it through Textile or Markdown
and Liquid converters, and spits out a complete, static website suitable for
serving with Apache or your favorite web server. This is also the engine
behind GitHub Pages, which you can use to host your project’s page or blog
right here from GitHub.</p>
<footer>— from the <a href="https://github.com/mojombo/jekyll/blob/master/README.textile">README</a></footer>
</blockquote>
<p>Just as the description says, each post is written in a
<a href="http://daringfireball.net/projects/markdown/">Markdown</a> formatted text file
with a special YAML front matter. Jekyll uses these content files and templates,
to produce static HTML pages. These pages are then served by the
<a href="http://nginx.org">nginx</a> web server.</p>
<p>Although some dynamic features such as comments (although they can be integrated
with services like Disqus) are not possible with the static Jekyll approach,
there are certain aspects that make up for these defiances: </p>
<ul>
<li>The content is available in a format that is future-proof and human-readable. </li>
<li>Jekyll-based blog hosting is extremely minimal. No PHP. No MySQL. Just a web
server and Jekyll running from time to time. Heck, it is even possible to host
your own Jekyll-based blog at GitHub.</li>
<li>Serving static pages is blazingly fast. Especially with a web-server built for
this purpose such as nginx.</li>
<li>It appeals to me as a nerd: I can hack it, I can manage and do things the way
I want them to happen, I can use <em>Vim</em> to edit the posts.</li>
</ul>
<h3 id="version-control-with-git">Version Control with Git</h3>
<p>The real fun is, that this blog is actually backed by Git. For each private
project that needs to be version controlled and available wherever I go, I add a
new Git repository to the <a href="https://github.com/sitaramc/gitolite/">Gitolite</a>
installation on my VPS. There is also one blog repository that contains the
Markdown sources, HTML templates, Jekyll configuration and assets. Whenever I
push new commits to the server, the Jekyll site-generator is called (see <a href="http://blog.zerosum.org/2010/11/01/pure-git-deploy-workflow.html">this
post</a> from
Nick Plante) and the resulting HTML files put in the directory, where nginx can
find them.</p>
<p>The usual workflow for me then goes like this:</p>
<ol>
<li>Create a new branch off of master that contains either a new post or a fix to
the layout.</li>
<li>Hack, hack, hack and commit.</li>
<li>If it is a long-term post, I will push the branch otherwise …</li>
<li>I merge with master with the branch and …</li>
<li>push the master.</li>
<li>The HTML will be re-generated and the new content goes online.</li>
</ol>
<p>Usually I setup a local Jekyll server using <code>jekyll --server</code> that serves on
<code>0.0.0.0:4000</code> to preview what I produce. This feels a bit like the
edit-pdflatex-view cycle when creating TeX documents but is a million times
faster than writing content into a shaky JavaScript editor of a Wordpress blog.</p>
<h3 id="responsive-design">Responsive Design</h3>
<p>Being a smartphone user, I always complain when web site developers do not
provide a specific mobile site or at least some CSS that adjusts the site to the
actual display device. Thus, I shamelessly stole the CSS from
<a href="http://87studios.net/psdtuts-2/a-simple-responsive-mobile-first-navigation/">87studios’</a>
responsive navigation tutorial and adapted it to my needs. In the following
screen shots you can see the difference:</p>
<p><div style="text-align: center;">
<img style="vertical-align:top;" src="/images/2012-07-28/2012-07-28-desktop.png">
<img style="vertical-align:top;" src="/images/2012-07-28/2012-07-28-mobile.png">
</div></p>
<p>This solution is not as fancy as <a href="http://mat-t.com/">Mat Tomaszewski’s</a> site, but
at least it does not look like total crap on mobile devices (that support media
queries).</p>
<h3 id="typography">Typography</h3>
<p>The main font used on this site is <a href="http://www.google.com/webfonts/specimen/PT+Sans">PT
Sans</a> by the Russian font
foundry Paratype. It has a large glyph coverage and is very legible, at least to
my eyes. Unfortunately, the kerning sometimes looks really bad when viewed
with Firefox. The code examples are set in <a href="http://www.google.com/webfonts/specimen/Droid+Sans+Mono">Droid Sans
Mono</a> designed for the
Android mobile phone OS.</p>
<h3 id="epilogue">Epilogue</h3>
<p>That are the technical details concerning this blog. Although the blog is working <em>as
advertised</em> I still have one problem: I am a terrible web site designer and have
only a very limited CSS knowledge. This is the reason why the <code>style.css</code> is
such a mess and the “archive” section at the bottom of the front page still does
not look like I want it to be. Hopefully, this will change in the future.</p>
<p>So, that’s it. If you feel you need more control over your content, you should
really consider to move to Jekyll and be part of the group of cool internet kids.</p>
<p>One last thing. You might ask what “bloerg” means: It doesn’t mean a thing.</p>
The Wonderful World of Heterogeneous Computing2012-07-19T00:00:00+00:002012-07-19T00:00:00+00:00
Unknown
https://bloerg.net/posts/heterogenous-computing/<p>There was once the promise of OpenCL, that all code written against the
specification is able to run on <em>any</em> device capable of running OpenCL code.
However, I was bitten by NVIDIA’s particular implementation and lost almost two
full days on figuring out what was the cause for a bug in our system.</p>
<p>After unsuccessful hours of tinkering with our own software, I thought the
problem might have to do with our heterogeneous system (at least in terms of
different GPUs: one GTX 680 and eight GTX 590s). I tried to replicate the bug
with a <a href="http://sscce.org/">SSCCE</a>, that is now available at
<a href="https://github.com/matze/ocl-regressions">GitHub</a>. When calling the regression
tool, you can specify the range of GPUs you want to include in the <code>cl_context</code>.</p>
<p>So, for the first GTX 680 I got:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ ./build/check --first=0 --last=0
</span><span># Platform: OpenCL 1.1 CUDA 5.0.1
</span><span># Device 0: GeForce GTX 680
</span><span>Creating kernel `two_const_params`: OK
</span><span>Creating kernel `three_const_params`: OK
</span><span>Creating kernel `two_local_params`: OK
</span><span>Creating kernel `three_local_params`: OK
</span><span>Creating kernel `two_global_params`: OK
</span><span>Creating kernel `three_global_params`: OK
</span></code></pre>
<p>Everything is fine. So let’s see what the eight GTX 580s do:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>$ ./build/check --first=1 --last=8
</span><span># Platform: OpenCL 1.1 CUDA 5.0.1
</span><span># Device 0: GeForce GTX 590
</span><span> ...
</span><span># Device 7: GeForce GTX 590
</span><span>Creating kernel `two_const_params`: OK
</span><span>Creating kernel `three_const_params`: OK
</span><span>Creating kernel `two_local_params`: OK
</span><span>Creating kernel `three_local_params`: OK
</span><span>Creating kernel `two_global_params`: OK
</span><span>Creating kernel `three_global_params`: OK
</span></code></pre>
<p>Again, no problems. Now, if I combine the GTX 680 with the first GTX 590, I get
this:</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>./build/check --first=0 --last=1
</span><span># Platform: OpenCL 1.1 CUDA 5.0.1
</span><span># Device 0: GeForce GTX 680
</span><span># Device 1: GeForce GTX 590
</span><span>Creating kernel `two_const_params`: OK
</span><span>Creating kernel `three_const_params`: Error: CL_INVALID_KERNEL_DEFINITION
</span><span>Creating kernel `two_local_params`: OK
</span><span>Creating kernel `three_local_params`: Error: CL_INVALID_KERNEL_DEFINITION
</span><span>Creating kernel `two_global_params`: OK
</span><span>Creating kernel `three_global_params`: OK
</span></code></pre>
<p>This is the problem I experienced and as you can see, it turned out that the
combination of a GTX 590 with a GTX 680 and kernels with more than two
<code>__constant</code> parameters triggered the bug.</p>
<p>As you can imagine, it was tricky to find the cause: On my system (two GTX 580s)
I never had any problems regardless of the number of <code>__constant</code> parameters.
There were also no problems on our development compute server, that has six GTX
580s. Moreover the error code returned by the system was not helpful at all. It
merely says that <code>clCreateKernel()</code> returns <code>CL_INVALID_KERNEL_DEFINITION</code> if</p>
<blockquote>
<p>the function definition for <code>__kernel</code> function given by [the] <em>kernel name</em>
such as the number of arguments, the argument types are not the same for all
devices for which the program executable has been built.</p>
<footer>— <a href="http://www.khronos.org/registry/cl/sdk/1.0/docs/man/xhtml/clCreateKernel.html">Khronos OpenCL 1.0 Specification</a></footer>
</blockquote>
<p>This is not case here. In fact, the error code is misleading as we always built
the kernels for all devices in one go, without changing the kernel function
signature for different devices.</p>
<p>Now, the real tragedy is, that one of my colleagues changed the GPUs on our
<em>production</em> compute server, without telling me and without running any
regression tests. Although this is not the cause of the bug, it really helped
not to detect it in the first place.</p>
<p>At the end of the day, I can just give you the advice to never trust the
implementation of a specification and to keep an eye on a changing environment,
be it hardware or software dependencies. Nevertheless, I hope this tool will be
helpful for others to check their particular environment.</p>
A Gtk+ 2D Point Viewer2012-07-15T00:00:00+00:002012-07-15T00:00:00+00:00
Unknown
https://bloerg.net/posts/gtk-point-viewer/<p>Ok, yet another widget. One of our cameras has an adjustable ADC gain that is
approximated by a piecewise linear function – that is, points connected with
lines. Because, I am the poor soul who should come up with a GUI to control this
camera, I decided to visualize this in an intuitive way rather than falling back
to boring spin buttons.</p>
<p>I remembered GIMP’s <a href="http://docs.gimp.org/de/gimp-tool-curves.html">curve tool</a> which is exactly what I had in mind, although
the widget’s purpose is a bit different. Therefore, I pulled out the useful bits
and mashed it together in the <code>EggPiecewiseLinearView</code> widget (yeah, what a name
…).</p>
<figure>
<img src="/images/2012-07-15/2012-07-15-pwl.png" />
</figure>
<p>At first, the view managed the data but I noticed soon, that this won’t scale
very well. I refactored the data point management into its own <code>EggDataPoints</code>
class just to recognize that this is also not exactly following the MVC pattern,
because in the perfect MVC world the model (in my case the data points) would
not control themselves. However, the Gtk people don’t seem to be active
proponents of the MVC pattern either: The <code>GtkAdjustment</code> and <code>GtkSpinButton</code>
relationship is pretty similar to <code>EggDataPoints</code> and <code>EggPiecewiseLinearView</code>.
So, after a short discussion with a friend, I though to myself: “Ok, it’s not
worth it, adding another <a href="http://geekandpoke.typepad.com/geekandpoke/2011/03/architectural-best-practices.html">layer</a>”.</p>
<p>Using the widget and the data store is self-explanatory and somewhat documented
by the <a href="https://github.com/matze/EggPiecewiseLinearView/blob/master/pwl-test.c">test program</a>.</p>
A Gtk+ Property Tree View2012-07-04T00:00:00+00:002012-07-04T00:00:00+00:00
Unknown
https://bloerg.net/posts/a-gtk-property-tree-view/<p>Recently, I faced the problem to display a <a href="http://developer.gnome.org/gtk/stable/GtkTreeView.html">tree
view</a> of properties. It
is trivial to setup a tree view that displays different kind of widgets (or
<a href="http://developer.gnome.org/gtk/stable/GtkCellRenderer.html">cell renderers</a> in
Gtk+ speak) per <em>column</em>. However, to properly interact with different property
types, I have to display a suitable cell renderer per <em>row</em>. This cannot be done
out-of-the-box, so I
<a href="http://stackoverflow.com/questions/11297187/in-gtk-is-it-possible-to-have-one-cell-renderer-per-treeview-row">consulted</a>
the excellent stackoverflow Q&A site. I was pointed to the <code>dconf-editor</code> that
does exactly this for the different key entries. Unfortunately, it is written in
Vala so I had to port it with my lacking Vala fluency. After some obstacles, I
came up with the <code>EggPropertyCellRenderer</code> and <code>EggPropertyTreeView</code> classes
that subclass from <code>GtkCellRenderer</code> and <code>GtkTreeView</code> respectively. You can
find the sources on <a href="https://github.com/matze/EggPropertyTreeView">GitHub</a>. Here
is a screenshot that shows you the tree view applied to the window itself.</p>
<figure>
<img src="/images/2012-07-04/2012-07-04-tree-view-test.png" />
</figure>
<p>The cell renderer is actually a royal hack. It subclasses from <code>GtkCellRenderer</code>
but does not provide any functionality on its own. Internally it creates spin
button, toggle, text and combo cell renderers to which rendering and sizing
requests are delegated. This is possible, because whenever a cell renderer is
about to be redrawn it is first updated with data using <code>g_object_set()</code>. At
this point, we know the property name, query it’s type using
<code>g_object_class_find_property()</code> and decide which renderer to use and how to
adapt it.</p>
<p>You can use it directly like so:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>GtkCellRenderer </span><span style="color:#72ab00;">*</span><span>renderer;
</span><span>GtkTreeViewColumn </span><span style="color:#72ab00;">*</span><span>column;
</span><span>
</span><span>renderer </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">egg_property_cell_renderer_new </span><span>(
</span><span> object, </span><span style="color:#7f8989;">/* a GObject whose properties you want to monitor */
</span><span> list_store </span><span style="color:#7f8989;">/* the model that must contain property names of 'object' in column 0 */
</span><span> );
</span><span>
</span><span>column </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">gtk_tree_view_column_new_with_attributes</span><span>(
</span><span> </span><span style="color:#d07711;">"Values"</span><span>, renderer,
</span><span> </span><span style="color:#d07711;">"prop-name"</span><span>, </span><span style="color:#b3933a;">0</span><span>,
</span><span> </span><span style="color:#b3933a;">NULL</span><span>);
</span><span>
</span><span style="color:#5597d6;">gtk_tree_view_append_column </span><span>(tree_view, column);
</span></code></pre>
<p>But you can also use the provided <code>EggPropertyTreeView</code> that encapsulates this
setup:</p>
<pre data-lang="cpp" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-cpp "><code class="language-cpp" data-lang="cpp"><span>GtkWidget </span><span style="color:#72ab00;">*</span><span>tree_view;
</span><span>
</span><span>tree_view </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">egg_property_tree_view_new </span><span>(object);
</span></code></pre>
<p>That’s it folks.</p>
Performance Impact of OpenCL Profiling2012-07-02T00:00:00+00:002012-07-02T00:00:00+00:00
Unknown
https://bloerg.net/posts/performance-impact-of-opencl-profiling/<p>Profiling an OpenCL program is mandatory for optimizing the kernel performance.
Timers have the inevitable drawback, that you cannot measure asynchronous kernel
calls reliably without inserting synchronous calls such as <code>clFinish()</code> or
<code>clFlush()</code>. This is usually not a problem, you would just put these into your
code and guard them with <code>#ifdef DEBUG/#endif</code>. You would lose the possibility
to do overlapping computation but this would not affect release builds.</p>
<p>However, in some cases you might want to measure the performance of your kernels
in production runs and still ensure asynchronous kernel execution. In this case,
you have to query the profiling information associated with event objects using
<code>clGetEventProfilingInfo()</code> and make sure to enable profiling of command queues
with the <code>CL_QUEUE_PROFILING_ENABLE</code> property when calling
<code>clCreateCommandQueue()</code>. Usually, these kind of things incur a performance
overhead. Because I wanted to know how big this impact would be, I wrote a small
<a href="https://github.com/matze/qperf">benchmark</a> utility which measures the run time
for two small kernels and different input sizes. I ran this benchmark on my home
computer that sports a NVIDIA GTX 480 and a Core i5 3450.</p>
<figure>
<img src="/images/2012-07-02/2012-07-02-simple-vs-intensive.png" />
</figure>
<p>As you can see for very small input sizes (less than 512×512 pixels), could have
a tremendous effect on the run-time. However, this also depends on the kind of
operation you execute. The simple kernel was merely calculating</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>output[tid] = input[tid] * 2.0f;
</span></code></pre>
<p>whereas the more computationally demanding kernel computed</p>
<pre style="background-color:#f5f5f5;color:#1f1f1f;"><code><span>output[tid] = cos(input[tid]) * exp(input[tid] * 2.0f);
</span></code></pre>
<p>Of course, relatively less time is spent managing the event objects when a more
complicated kernel is in place. This is even more so true, if you increase the
number of computed elements. With more than 512×512 elements, the overhead
becomes negligible and is sometimes even negative.</p>
<p>As a conclusion, you could do run-time evaluation for whatever reason, if your
kernels are computationally demanding (which they are anyway, right?).</p>
Customizing the Frametitle of a Beamer Presentation2012-06-21T00:00:00+00:002012-06-21T00:00:00+00:00
Unknown
https://bloerg.net/posts/customizing-the-frametitle-of-beamer-presentation/<p><a href="http://www.ctan.org/tex-archive/macros/latex/contrib/beamer/">Beamer</a> is a
package, written by <a href="http://www.tcs.uni-luebeck.de/mitarbeiter/tantau/">Till
Tantau</a>, to create
presentation slides within the TeX eco system. It is associated with some
scientific communities such CS and Math and comes with all benefits and
disadvantages a TeX package could have: superior input format, nice
typographical output but the default themes are somewhat lacking the <em>kick</em> and
customization is a royal PITA.</p>
<p>For me the question was: how hard could it be to create a theme that has a
colored frame title bar, the title with a nice font on the left and a logo on
the left, just like this:</p>
<p><img src="/images/2012-06-21/2012-06-21-frametitle.png" alt="Customized frame title" /></p>
<p>Not that hard actually but it takes quite some time to understand the ideas and
tweak things until it looks just alright. Unfortunately, there is virtually no
information on <em>how</em> to customize themes except by studying existing themes.</p>
<p>From the beamer guide you can easily guess that the different inner, outer, etc.
templates are an orthogonal way to specify the look of your presentation. In
order to change the appearance of the frame title, we have to create a new style
file called <code>beamerouterthemefoo.sty</code> that is load inside your TeX file via
<code>\useoutertheme{foo}</code>. When you have a close look at the pre-load style
<code>beamerouterthemedefault.sty</code>, you will notice that the template for the frame
title is already defined. If you redefine it, hell breaks loose, so we <em>have to
set</em> our new style. Let’s start simple and just insert the current frame title:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\setbeamertemplate</span><span>{frametitle}
</span><span>{
</span><span> </span><span style="color:#b39f04;">\insertframetitle
</span><span>}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-1.png" alt="Default frame title" /></p>
<p>The result is not surprising: bland default color (if you haven’t changed it)
and default placement. Let’s add a color box, so that the title is separated
from the actual content:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\setbeamertemplate</span><span>{frametitle}
</span><span>{
</span><span> </span><span style="color:#72ab00;">\begin</span><span>{</span><span style="color:#5597d6;">beamercolorbox</span><span>}{frametitle}
</span><span> </span><span style="color:#b39f04;">\insertframetitle
</span><span> </span><span style="color:#72ab00;">\end</span><span>{</span><span style="color:#5597d6;">beamercolorbox</span><span>}
</span><span>}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-2.png" alt="Colorized frame title" /></p>
<p>Hmm, nothing changed except for position of the text. Notice the parameter to
the <code>colorbox</code>. It is used to determine font and color properties of this color
box. Up to now, this hasn’t changed. So, let’s create a new
<code>beamercolorthemefoo.sty</code> file and define some colors that we load with
<code>\usecolortheme{foo}</code>:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\definecolor</span><span>{greyone}{RGB}{77,77,77}
</span><span style="color:#b39f04;">\setbeamercolor</span><span>{palette quaternary}{fg=white,bg=greyone}
</span><span style="color:#b39f04;">\setbeamercolor</span><span>{titlelike}{parent=palette quaternary}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-3.png" alt="Really colorized frame title" /></p>
<p>Not bad but the color box is only as wide as the regular content width and only
as high as the text. We can fix this with the <code>ht</code> and <code>wd</code> parameters:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#72ab00;">\begin</span><span>{</span><span style="color:#5597d6;">beamercolorbox</span><span>}[</span><span style="color:#5597d6;">ht</span><span>=1.8em,wd=</span><span style="color:#b39f04;">\paperwidth</span><span>]{frametitle}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-4.png" alt="Correct box size" /></p>
<p>Apart from the alignment this looks good now. However, you will notice a glitch
as soon as you have a title that contains descenders in its letters. For
example, the ‘Q’ in the next slide reaches below the base line thus shifting the
whole text slightly upwards:</p>
<p><img src="/images/2012-06-21/2012-06-21-wip-5.png" alt="A customized frametitle" /></p>
<p>The LaTeX answer to this is the <code>\strut</code> command that inserts invisible elements
that ensure a guaranteed height:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\strut\insertframetitle\strut
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-6.png" alt="A customized frametitle" /></p>
<p>Now it gets tricky. First of all, you notice a small gap between the color box
and the top margin. The default theme styles fix this with a <code>\nointerlineskip</code>
– a not-so-obvious trick. To center the frame title, we need to adjust the
<code>sep</code> parameter of the color box (kind of like a padding that originates from
TikZ) and make some vertical room with <code>\vskip</code> until it fits somehow. This is
ugly and one of the reasons I have a hard time with customizing Beamer themes:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\setbeamertemplate</span><span>{frametitle}
</span><span>{
</span><span> </span><span style="color:#b39f04;">\nointerlineskip
</span><span> </span><span style="color:#72ab00;">\begin</span><span>{</span><span style="color:#5597d6;">beamercolorbox</span><span>}[</span><span style="color:#5597d6;">sep</span><span>=0.3cm,ht=1.8em,wd=</span><span style="color:#b39f04;">\paperwidth</span><span>]{frametitle}
</span><span> </span><span style="color:#b39f04;">\vbox</span><span>{}</span><span style="color:#b39f04;">\vskip</span><span>-2ex</span><span style="color:#7f8989;">%
</span><span> </span><span style="color:#b39f04;">\strut\insertframetitle\strut
</span><span> </span><span style="color:#b39f04;">\vskip</span><span>-0.8ex</span><span style="color:#7f8989;">%
</span><span> </span><span style="color:#72ab00;">\end</span><span>{</span><span style="color:#5597d6;">beamercolorbox</span><span>}
</span><span>}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-7.png" alt="A customized frametitle" /></p>
<p>Ok, things get in shape but I also want our university’s logo on the right side.
Thus, I experimented with different kind of boxes and minipages to right align
it, but all attempts utterly failed. Until I just tried to fill the horizontal
space via <code>\hfill</code> et voilà …</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\strut\insertframetitle\strut
</span><span style="color:#b39f04;">\hfill
</span><span style="color:#b39f04;">\includegraphics</span><span>[</span><span style="color:#5597d6;">width</span><span>=1cm]{logo}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-8.png" alt="A customized frametitle" /></p>
<p>… merde. Although the logo’s content has no inner spacing it is shifted
slightly upwards. We take out the hammer and use a <code>\raisebox</code> to shift it down until
it looks alright:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#b39f04;">\raisebox</span><span>{-0.8mm}{</span><span style="color:#b39f04;">\includegraphics</span><span>[</span><span style="color:#5597d6;">width</span><span>=1cm]{logo}}
</span></code></pre>
<p>Moreover, I wanted to use a font that is a) not CM sans again and b) fits more
to the logo. I decided to use the free bold Droid Sans typeface. Because I
didn’t want to spend any more time on LaTeX pecularities, I went the easy way
and let XeLaTeX turn the TeX file into a PDF. For the font support, we simply
drop the following in the preamble:</p>
<pre data-lang="latex" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-latex "><code class="language-latex" data-lang="latex"><span style="color:#72ab00;">\usepackage</span><span>{</span><span style="color:#a2a001;">xltxtra</span><span>}
</span><span style="color:#72ab00;">\usepackage</span><span>{</span><span style="color:#a2a001;">polyglossia</span><span>}
</span><span style="color:#b39f04;">\setsansfont</span><span>[BoldFont={Droid Sans Bold}]{Droid Sans}
</span><span>
</span><span style="color:#b39f04;">\setbeamerfont</span><span>{frametitle}{size=</span><span style="color:#b39f04;">\Large</span><span>,series=</span><span style="color:#b39f04;">\bfseries</span><span>}
</span></code></pre>
<p><img src="/images/2012-06-21/2012-06-21-wip-9.png" alt="A customized frametitle" /></p>
<p>Here we go. Nicely laid out. No skipping and shifting when switching slides. But
using XeTeX comes with a twist: it does not work with Beamers built-in slide
transitions …</p>
<p><em>Update</em>: There is some more in-depth information about frame title
customization over at
<a href="http://tex.stackexchange.com/questions/38208/beamer-create-own-headline-theme">tex.stackexchange</a>.</p>
Waf and OpenCL check2012-06-10T00:00:00+00:002012-06-10T00:00:00+00:00
Unknown
https://bloerg.net/posts/waf-and-opencl-check/<p>Although I am quite used to CMake and the infamous Autotools, I wanted to try
out the <a href="http://code.google.com/p/waf/">Waf</a> build system for a smallish side
project that I am investigating with a fellow student of mine. This project has
a limited number of dependencies which could also be integrated in a simple
<code>Makefile</code> but it also needs to detect the include path and the libraries of any
OpenCL installation. Unfortunately, neither NVIDIA’s nor AMD’s OpenCL
distribution is installed in a standards compliant way. Although the NVIDIA
installation procedure suggests to install the CUDA toolkit into <code>/usr/local</code>,
it does so by creating a new root folder <code>/usr/local/cuda</code> with <code>bin</code>, <code>lib</code>,
<code>include</code> and a bunch of non-UNIX directories right beneath. But I don’t want to
complain, let’s head right in how to solve this.</p>
<p>Download the latest Waf distribution file and create an empty <code>wscript</code> file in
your source directory. This file is used to configure – check if required
and optional dependencies are met – and build your project. Because it contains
real <em>Python</em> code, you can configure your project in any imaginable way.</p>
<p>The configuration and build steps are mapped to the <code>configure()</code> and <code>build()</code>
functions. To compile C source files, we need to tell the system which
translation facility to use:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">configure</span><span>(</span><span style="color:#5597d6;">conf</span><span>):
</span><span> conf.</span><span style="color:#5597d6;">load</span><span>(</span><span style="color:#d07711;">'compiler_c'</span><span>)
</span><span> conf.env.</span><span style="color:#5597d6;">append_unique</span><span>(</span><span style="color:#d07711;">'CFLAGS'</span><span>, [</span><span style="color:#d07711;">'-g'</span><span>, </span><span style="color:#d07711;">'-std=c99'</span><span>, </span><span style="color:#d07711;">'-O3'</span><span>, </span><span style="color:#d07711;">'-Wall'</span><span>, </span><span style="color:#d07711;">'-Werror' </span><span>])
</span><span>
</span><span style="color:#72ab00;">def </span><span style="color:#c23f31;">build</span><span>(</span><span style="color:#5597d6;">bld</span><span>):
</span><span> bld.</span><span style="color:#5597d6;">program</span><span>(</span><span style="color:#5597d6;">source</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'foo.c'</span><span>, </span><span style="color:#5597d6;">target</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'foo'</span><span>)
</span></code></pre>
<p>As you can see, additional <code>CFLAGS</code> are appended through the <code>env</code> objects
<code>append_x</code> methods. For some reason, we must also load the compiler in a
preceding options step, which tells the build system to export the compiler’s
command line options:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">options</span><span>(</span><span style="color:#5597d6;">opt</span><span>):
</span><span> opt.</span><span style="color:#5597d6;">load</span><span>(</span><span style="color:#d07711;">'compiler_c'</span><span>)
</span></code></pre>
<p>Most Linux distributions ship
<a href="http://en.wikipedia.org/wiki/Pkg-config">pkg-config</a> files for the majority of
libraries. Fortunately, Waf is able to call the pkg-config binary out-of-the-box
with the <code>check_cfg</code> function.</p>
<p>However, the NVIDIA installer copies the header files in non-standard locations
and does not provide any <code>.pc</code> files. In most cases, the user will just hit the
enter key when prompted by the installer where to put the files, thus copying it
to <code>/usr/local/cuda</code>. Others, like me, try to keep non-standard things in <code>/opt</code>
or place the distribution in their home directory and set the suggested in
environment variables. To cover these cases, we can compute a list with existing
paths:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">guess_cl_include_path</span><span>():
</span><span> </span><span style="color:#72ab00;">import </span><span>os
</span><span>
</span><span> </span><span style="color:#5597d6;">OPENCL_INC_PATHS </span><span style="color:#72ab00;">= </span><span>[
</span><span> </span><span style="color:#d07711;">'/usr/local/cuda/include'</span><span>,
</span><span> </span><span style="color:#d07711;">'/opt/cuda/include'
</span><span> ]
</span><span>
</span><span> </span><span style="color:#72ab00;">try</span><span>:
</span><span> OPENCL_INC_PATHS.</span><span style="color:#5597d6;">append</span><span>(os.environ[</span><span style="color:#d07711;">'CUDA_INC_PATH'</span><span>])
</span><span> </span><span style="color:#72ab00;">except</span><span>:
</span><span> </span><span style="color:#72ab00;">pass
</span><span>
</span><span> </span><span style="color:#72ab00;">return </span><span style="color:#b39f04;">filter</span><span>(</span><span style="color:#72ab00;">lambda </span><span style="color:#5597d6;">d</span><span>: os.path.</span><span style="color:#5597d6;">exists</span><span>(d), </span><span style="color:#5597d6;">OPENCL_INC_PATHS</span><span>)
</span></code></pre>
<p>We plug this little function into the configure function, tell the user what’s
going on with <code>start_msg()</code> and abort with <code>fatal()</code> in case we cannot find
anything. Last but not least, we add a check with <code>check_cc()</code> for
<code>libOpenCL.so</code> which should be installed in one of the standard library paths.
The final configure step looks like this:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">configure</span><span>(</span><span style="color:#5597d6;">conf</span><span>):
</span><span> conf.</span><span style="color:#5597d6;">load</span><span>(</span><span style="color:#d07711;">'compiler_c'</span><span>)
</span><span> conf.env.</span><span style="color:#5597d6;">append_unique</span><span>(</span><span style="color:#d07711;">'CFLAGS'</span><span>, [</span><span style="color:#d07711;">'-g'</span><span>, </span><span style="color:#d07711;">'-std=c99'</span><span>, </span><span style="color:#d07711;">'-O3'</span><span>, </span><span style="color:#d07711;">'-Wall'</span><span>, </span><span style="color:#d07711;">'-Werror' </span><span>])
</span><span>
</span><span> </span><span style="color:#7f8989;"># use pkg-config
</span><span> conf.</span><span style="color:#5597d6;">check_cfg</span><span>(</span><span style="color:#5597d6;">package</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'glib-2.0'</span><span>, </span><span style="color:#5597d6;">args</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'--cflags --libs'</span><span>, </span><span style="color:#5597d6;">uselib_store</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'GLIB2'</span><span>)
</span><span>
</span><span> conf.</span><span style="color:#5597d6;">start_msg</span><span>(</span><span style="color:#d07711;">'Checking for OpenCL include path'</span><span>)
</span><span> incs </span><span style="color:#72ab00;">= </span><span style="color:#5597d6;">guess_cl_include_path</span><span>()
</span><span>
</span><span> </span><span style="color:#72ab00;">if </span><span>incs:
</span><span> conf.env.</span><span style="color:#5597d6;">OPENCL_INC_PATH </span><span style="color:#72ab00;">= </span><span>incs[</span><span style="color:#b3933a;">0</span><span>]
</span><span> conf.</span><span style="color:#5597d6;">end_msg</span><span>(</span><span style="color:#d07711;">'found'</span><span>)
</span><span> </span><span style="color:#72ab00;">else</span><span>:
</span><span> conf.</span><span style="color:#5597d6;">fatal</span><span>(</span><span style="color:#d07711;">'OpenCL include path not found'</span><span>)
</span><span>
</span><span> conf.</span><span style="color:#5597d6;">check_cc</span><span>(</span><span style="color:#5597d6;">lib</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'OpenCL'</span><span>, </span><span style="color:#5597d6;">uselib_store</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'CL'</span><span>)
</span></code></pre>
<p>Configure information is stored across builds using the <code>env</code> structure and the
<code>uselib_store</code> keyword. When building the binary, we refer to the these
variables and we are good to go:</p>
<pre data-lang="python" style="background-color:#f5f5f5;color:#1f1f1f;" class="language-python "><code class="language-python" data-lang="python"><span style="color:#72ab00;">def </span><span style="color:#c23f31;">build</span><span>(</span><span style="color:#5597d6;">bld</span><span>):
</span><span> bld.</span><span style="color:#5597d6;">program</span><span>(</span><span style="color:#5597d6;">source</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'foo.c'</span><span>,
</span><span> </span><span style="color:#5597d6;">target</span><span style="color:#72ab00;">=</span><span style="color:#d07711;">'foo'</span><span>,
</span><span> </span><span style="color:#5597d6;">use</span><span style="color:#72ab00;">=</span><span>[</span><span style="color:#d07711;">'GLIB2'</span><span>, </span><span style="color:#d07711;">'CL'</span><span>],
</span><span> </span><span style="color:#5597d6;">includes</span><span style="color:#72ab00;">=</span><span>bld.env.</span><span style="color:#5597d6;">OPENCL_INC_PATH</span><span>)
</span></code></pre>