Go to content Go to navigation

ZetaWatch for Developers · 2019-05-05 17:37 by Black in

ZFS Interaction

ZetaWatch communicates with zfs using libzfs.dylib, libzfs_core.dylib ,
libzpool.dylib and libnvpair.dylib, just like the command line tools do. This gives
it all the flexibility of the command line tools, at the cost of having to reimplement
functionality that is found in the tools and not the library. And since the libraries are
explicitly not meant to provide a stable ABI, ZetaWatch is also closely coupled to the
ZFS version it is built and written for.

All the ZFS interaction is wrapped in the ZFSWrapper library. This C++ library isolates
the issues mentioned above and provides a more convenient and safe API than the original
C interface does. The library is used both by the helper tool and the frontend app. This
is the most reusable part of ZetaWatch, and might be split out as separate project later.

ZFSUtils
contains most of the advanced functionality, such as C++ Wrappers around the
library, pool, vdev or file system handles. Those classes also have functionality to query
state and iterate over members.
ZFSNVList
provides a wrapper around the nvpair_t / nvlist_t data structure that
is used in ZFS for a lot of userland / kernel communication. It manages resources in both
owning and non-owning fashion, and allows for easier iteration over sequences.
ZFSStrings
translate ZFS status enums into the user facing emoji or string
description, optionally with localization. (Localization is not well tested or supported
at the moment.)

Helper Tool

The implementation of the helper tool follows apple’s EvenBetterAuthorizationSample

The helper tool communicates with the user application via AuthorizationService and
NSXPCConnection. The application side of code for this is in ZetaAuthorization.m. The
RPC protocol can be found in ZetaAuthorizationHelperProtocol.h, and is implemented in
ZetaAuthorizationHelper.mm. The CommonAuthorization.m file contains the supported
commands and associated default permissions.

The helper tool can be uninstalled with the `uninstall-helper.sh` script. This is useful
for debugging the installation of the helper, or updating the helper without increasing
the bundle version.

Security & Code Signing

Official release builds are signed and notarized, and should run without issues even on
newer Mac OS X. But there are still issues with authentication reported with the program
not being recognized as signed. To verify security manually, the following commands can
be used:

codesign -v -v -d ZetaWatch.app
xcrun stapler validate -v ZetaWatch.app

Building ZetaWatch requires an apple developer account with DeveloperID signing
capabilities, since it uses SMJobBless to run a helper service as root. This service
executes actions on behalf of the user, such as mounting, unmounting or loading a key.
Notarization is required to create binaries that can be run without without warning on
the newest Mac OS X.

Comment

ZetaWatch · 2019-05-05 01:54 by Black in

ZetaWatch is a small OS X program that displays the zfs status in the menu bar, similar to what iStat Menus does for other information. It is fairly well tested, but due to the current state of libzfs and libzfs_core, changes will be required until the API
stabilizes. ZetaWatch is usually compiled for the latest available ZFS release for Mac OS, and might not be compatible with other releases.

Currently supported features are:

  • Show pool and vdev status including scrub progress
  • Show pool / filesystem properties
  • Show filesystem and vdev stats
  • Import pools and Export pools
  • Mount / unmount datasets
  • Load encryption keys for encrypted datasets
  • Scrub pools
  • Report errors in notification center

Check it out :)

Comment

CaptureOne for Sony · 2019-03-31 14:54 by Black in

Apple’s Photos isn’t too great, Adobe’s Lightroom requries a subscription, Sony’s Edit is quite lacking in feature. After some searching around, I settled for PhaseOne’s Capture One, and it’s free, for Sony (or Fujifilm) cameras.

Some features are only available in the pro version, which is available as pay-once or subscription, but the free version is already very feature rich, and includes asset management, raw editing and more. And it’s quite easy to use.

This picture was taken under dim sodium light, handheld, with ISO 6400, and edited slightly (and not very skillfully) in CaptureOne Express.

Here a 1:1 cutout of the original, converted to JPEG with Apple’s Preview.

One click to adjust white balance, a bit of playing with auto-exposure and the curves to change the light, and adjust noise reduction a bit.

Comment

High Resolution Photography · 2019-03-31 13:53 by Black in

I’ve been playing around with Photography as a hobby for years, but never mustered enough motivation to actually take pictures regularly. But recently I bought a Sony A7Rii, a 42 MPix full frame Sony E-Mount mirrorless camera, To take cat pictures. Compensating for lack of experience with superior technology (despite its age, the camera is still great), seems to work :)

The resolution of the sensor is incredible. And with the Sony 85mm/F1.8 lens, boke is easy to get.

Maybe too easy, the depth of field wide open for close subjects is only a few cm. And at F1.8, color fringing / can be quite noticeable.

I really like this lens. For wide angle pictures, I own a Samyang 24mm/F2.8, but for the kind of pictures I take (portraits of cats), it is much less suitable.

My current gear:

Comment

Time flies · 2019-03-24 11:59 by Black in

After finishing my masters thesis and completing civil service, I’ve been working for Disney Research since early 2010. And I’ve obviously not had much to say on this blog. Most of my early work was focused on software engineering, mostly tech transfer using C++. I’ve written the basis of a big code base that is used (and considering the resistance to update anything will likely be used for decades) in many projector / camera systems in disney parks. Qt was popular for GUI work back then, and I’ve used it in several projects targeted at normal humans (but since those were internal research projects, not many actually used it). I’ve worked on projects targeting mobile platforms, in both Objective-C and Unity (I’m even named on a patent for one of them). I’ve written plugins for Nuke, ToonBoom, AfterEffects and many others, code running on arduinos or servers with dozens of cores and multiple GPUs.

But lately, I’ve shifted to projects that focus on machine learning. I’m not a researcher, so I don’t focus on developing models and graphs, but over the past few years I’ve debugged, ported and improved a lot of the deep learning research code created here. My favorite framework is TensorFlow, and I spent most time with it, but I’ve also used PyTorch. TensorFlow’s graph based structure forces more discipline, and makes reasoning much easier than the python spaghetti code I’ve seen from torch users. The biggest NN project I’ve participated in so far was Denoising with Kernel Prediction. Implemented in TensorFlow, and using custom CUDA code for better performance, this denoiser surpasses anything before.

I still like C++, it is one of the most flexible and pragmatic languages that are widely used. But for deep learning, Python is the standard, and inertia ensures this will remain the case for many years. Which is unfortunate, the lack of type safety and static checking is a big annoyance, especially in non-trivial code bases as the ones I work in.
Using C++ for TensorFlow is possible but only done for very specific subset of tasks, such as embedding into other programs, or writing custom Ops with CPU or CUDA.

Comment

SVG embedding in XHTML with TextPattern · 2010-02-10 19:20 by Black in

To store and transport scale independent graphics SVG is a standardised format. It is supported by most browsers somewhat, with the exception of Microsoft’s product. It would be nice if it could be used like the usual raster image formats in an <img /> tag. While Safari and other WebKit browsers support this usage, Firefox does not. Instead, SVG can be embedded directly into the source code of an XHTML file.

45° A B C

The above image is part of the source code of this page. It is included by a TextPattern plugin, and only slightly processed. Processing is needed to remove the <?xml /> header and to insert a viewBox attribute to allow scaling of the image with CSS. With that done, Firefox displays and scales the image nicely. Safari on the other hand causes trouble, it does not correctly infer the viewport height from the width and the aspect ratio.

To be allowed to embed SVG data, the mime type of the document has to be application/xhtml+xml or similar. This has to be changed for TextPattern by editing the header() call in publish.php. The plugin code itself is rather simple. Download a version ready to be pasted into TextPattern (Licensed under the MIT). Below the sourcecode.

svg_inline.php [1.74 kB]

  1. function svg_inline($atts)
  2. {
  3.   extract(lAtts(array(
  4.     'src'  => '',
  5.   ),$atts));
  6.  
  7.   if ($src)
  8.   {
  9.     if ($src[0] == '/')
  10.     {
  11.       // Relative to Document Root
  12.       $src = $_SERVER['DOCUMENT_ROOT'].$src;
  13.     }
  14.     $svg = file_get_contents($src, FILE_TEXT);
  15.     if ($svg)
  16.     {
  17.       // Add this to publish.php
  18.       //header("Content-type: application/xhtml+xml; charset=utf-8");
  19.       $svg = preg_replace('/<\?xml [^>]*>/', '', $svg, 1);
  20.       $svg = preg_replace('/(<svg[^>]*)width="([^"]*)"([^>]*)height="([^"]*)"([^>]*)>/',
  21.         '$1$3$5 viewBox="0 0 $2 $4">', $svg, 1);
  22.       return '<div class="svg">'.$svg.'</div>';
  23.     }
  24.     else
  25.     {
  26.       return 'Read error src='.$src;
  27.     }
  28.   }
  29.   else
  30.   {
  31.     return 'Missing src';
  32.   }
  33. }

TextPattern plugins are php functions that take two arguments: an array containing the tag attributes, and the contents of the tag element. All this plugin function does is to read the specified svg file and return the filtered source. A simple <txp:svg_inline src="imagepath" /> results in a nicely embedded SVG.

Comment