Go to content Go to navigation

Lua String Writer · 2010-02-04 16:35 by Black in

Lua strings are opaque byte streams. They are constant, and can only be manipulated by using the string api to create new strings. This can be expensive, especially when creating a string by appending new values at the end. While Lua contains optimizations for direct concatenation, successive appending has a high overhead.

This StringWriter class reduces the overhead by aggregating string concatenations in a table and executing them when requested. It was originally designed to serve as an efficient drop-in replacement for files as created by io.open, but it can also be used standalone.

The class itself is built with a protected shared metatable and state inside a table. The state itself is not protected (it would be possible by using individual metatables or an internal database in a weak table, but this is more elegant). The metatable contains entries to redirect reads to the method table, redirect new writes to nothing and prevent changing or reading the metatable. The concatenation operator is also overloaded, but since it has value semantic, and is not allowed to change the object itself, the implementation is less efficient than StringWriter:write(). Converting a StringWriter with tostring() gives the contained string, equivalent to StringWriter:get().

stringwriter.lua [4.77 kB]

  1. -- MetaTable for string writers
  2. local StringWriter_Meta = {
  3.   ["__index"] = StringWriter_Methods;
  4.   ["__newindex"] = function ()
  5.       -- Don't allow setting values
  6.     end;
  7.   ["__metatable"] = StringWriter_ID;
  8.   ["__tostring"] = StringWriter_Methods.get;
  9.   ["__concat"] = function (this, str)
  10.       str = tostring(str);
  11.       local sw = StringWriter();
  12.       sw.string_ = {};
  13.       for _, v in ipairs(this.string_) do
  14.         table.insert(sw.string_, v);
  15.       end
  16.       table.insert(sw.string_, str);
  17.       sw.len_ = this.len_ + #str;
  18.       sw.pos_ = sw.len_;
  19.       return sw;
  20.     end;
  21. }

The method table itself contains all methods the StringWriter supports. It was modeled after the file class, so many methods are placeholders that do nothing. The methods that are supported are seeking and writing. Seeking simply sets an internal position value. Writing in the context of files means overwriting and extending. When the position is at the end, the contents that are to be written can simply be appended to the contents table. Otherwise, the string has to be baked, split, and recomposed.

stringwriter.lua [4.77 kB]

  1. -- Methods for string writers
  2. local StringWriter_Methods = {
  3.   ["close"] = voidFunc;
  4.   ["flush"] = voidFunc;
  5.   ["lines"] = voidFunc;
  6.   ["read"] = voidFunc;
  7.   ["seek"] = function (this, base, offset)
  8.       -- Only act on StringWriters
  9.       if not StringWriter_Check(this) then
  10.         return nil, "Invalid StringWriter";
  11.       end;
  12.       -- Default offset
  13.       if type(base) == "number" then
  14.         offset = base; -- Not done in file, but reasonable
  15.       else
  16.         offset = offset or 0;
  17.       end
  18.       -- Set position and return it
  19.       if base == "set" then
  20.         this.pos_ = math.clamp(offset,0, this.len_);
  21.       elseif base == "end" then
  22.         this.pos_ = math.clamp(#this.string_+offset,0, this.len_);
  23.       else -- "cur"
  24.         this.pos_ = math.clamp(this.pos_+offset,0, this.len_);
  25.       end
  26.       return this.pos_;
  27.     end;
  28.   ["setvbuf"] = voidFunc;
  29.   ["write"] = function (this, ...)
  30.       -- Only act on StringWriters
  31.       if not StringWriter_Check(this) then return end;
  32.       -- Concat all arguments (assuming they are valid)
  33.       local s = table.concat({...});
  34.       -- Concat argument string with current string
  35.       if this.pos_ == -1 or this.pos_ == this.len_ then
  36.         -- Just append
  37.         table.insert(this.string_, s);
  38.       else
  39.         -- Insert, merge into a string
  40.         local sFull = table.concat(this.string_);
  41.         -- Split it up
  42.         local sLeft = string.sub(sFull, 1, this.pos_);
  43.         local sRight = string.sub(sFull, this.pos_+1+#s, -1)
  44.         -- And put it back in
  45.         this.string_ = {sLeft, s, sRight};
  46.       end
  47.       -- Update position
  48.       this.pos_ = this.pos_ + #s;
  49.       if this.pos_ > this.len_ then
  50.         this.len_ = this.pos_;
  51.       end;
  52.     end;
  53.   ["get"] = function (this)
  54.       if not StringWriter_Check(this) then
  55.         return nil, "Invalid StringWriter";
  56.       else
  57.         this.string_ = {table.concat(this.string_)};
  58.         return this.string_[1];
  59.       end;
  60.     end;
  61. }

StringWriter instances are created by a factory method. It initializes the state and sets the metatable.

stringwriter.lua [4.77 kB]

  1. -- StringWriter factory
  2. StringWriter = function ()
  3.   local sw = {
  4.     string_ = {""};
  5.     len_  = 0;
  6.     pos_  = 0;
  7.   }
  8.   setmetatable(sw, StringWriter_Meta);
  9.   return sw;
  10. end

I hope this code is useful for someone, use it as you wish, it is licensed under the MIT license.


Lua Table Persistence · 2010-01-27 14:56 by Black in

Lua is a very flexible scripting language for embedding into programs. It’s standard API is very slim, it lacks all but basic functions. Adding them is easy though.

The persistence code here requires nothing but lua’s standard io.open for reading and writing files. It can handle loops, multiple references to the same table in both keys and values, and most standard value types.
Not supported are userdata, threads and many types of functions. Exporting simple lua functions works, but the exported byte code is not portable. The result from the export is itself lua code, it can be executed and returns data structures equivalent to those that were exported.

The core for the export is a simple recursion with a dispatcher method and writers for all types. When unsupported types are encountered, nil is written. This can cause problems on import when those unsupported values are used as table keys, but in most cases it is more desirable than to fail the export.

persistence.lua [5.50 kB]

  1. -- Format items for the purpose of restoring
  2. writers = {
  3.   ["nil"] = function (file, item)
  4.       file:write("nil");
  5.     end;
  6.   ["number"] = function (file, item)
  7.       file:write(tostring(item));
  8.     end;
  9.   ["string"] = function (file, item)
  10.       file:write(string.format("%q", item));
  11.     end;
  12.   ["boolean"] = function (file, item)
  13.       if item then
  14.         file:write("true");
  15.       else
  16.         file:write("false");
  17.       end
  18.     end;
  19.   ["table"] = function (file, item, level, objRefNames)
  20.       local refIdx = objRefNames[item];
  21.       if refIdx then
  22.         -- Table with multiple references
  23.         file:write("multiRefObjects["..refIdx.."]");
  24.       else
  25.         -- Single use table
  26.         file:write("{\n");
  27.         for k, v in pairs(item) do
  28.           writeIndent(file, level+1);
  29.           file:write("[");
  30.           write(file, k, level+1, objRefNames);
  31.           file:write("] = ");
  32.           write(file, v, level+1, objRefNames);
  33.           file:write(";\n");
  34.         end
  35.         writeIndent(file, level);
  36.         file:write("}");
  37.       end;
  38.     end;
  39.   ["function"] = function (file, item)
  40.       -- Does only work for "normal" functions, not those
  41.       -- with upvalues or c functions
  42.       local dInfo = debug.getinfo(item, "uS");
  43.       if dInfo.nups > 0 then
  44.         file:write("nil --[[functions with upvalue not supported]]");
  45.       elseif dInfo.what ~= "Lua" then
  46.         file:write("nil --[[non-lua function not supported]]");
  47.       else
  48.         local r, s = pcall(string.dump,item);
  49.         if r then
  50.           file:write(string.format("loadstring(%q)", s));
  51.         else
  52.           file:write("nil --[[function could not be dumped]]");
  53.         end
  54.       end
  55.     end;
  56.   ["thread"] = function (file, item)
  57.       file:write("nil --[[thread]]\n");
  58.     end;
  59.   ["userdata"] = function (file, item)
  60.       file:write("nil --[[userdata]]\n");
  61.     end;
  62. }

To be able to export tables that are referenced several times (be it a cycle in the data structure, or just one that is inserted several times), the structures that are to be written are examined first and the numbers or references to each table are counted.

All tables that have multiple references to them are created at the start in the export file before they are filled with content. This is required, since they could contain themselves or other multi-ref tables.

After all those temporary tables are created, they are filled with content. The writer for tables uses a lookup table for multi-ref tables, instead of creating the table constructor for them, they are assigned from the table created at the start. Last but not least, the passed arguments themselves are created in the same way.

persistence.lua [5.50 kB]

  1.   store = function (path, ...)
  2.     local file, e;
  3.     if type(path) == "string" then
  4.       -- Path, open a file
  5.       file, e = io.open(path, "w");
  6.       if not file then
  7.         return error(e);
  8.       end
  9.     else
  10.       -- Just treat it as file
  11.       file = path;
  12.     end
  13.     local n = select("#", ...);
  14.     -- Count references
  15.     local objRefCount = {}; -- Stores reference that will be exported
  16.     for i = 1, n do
  17.       refCount(objRefCount, (select(i,...)));
  18.     end;
  19.     -- Export Objects with more than one ref and assign name
  20.     -- First, create empty tables for each
  21.     local objRefNames = {};
  22.     local objRefIdx = 0;
  23.     file:write("-- Persistent Data\n");
  24.     file:write("local multiRefObjects = {\n");
  25.     for obj, count in pairs(objRefCount) do
  26.       if count > 1 then
  27.         objRefIdx = objRefIdx + 1;
  28.         objRefNames[obj] = objRefIdx;
  29.         file:write("{};"); -- table objRefIdx
  30.       end;
  31.     end;
  32.     file:write("\n} -- multiRefObjects\n");
  33.     -- Then fill them (this requires all empty multiRefObjects to exist)
  34.     for obj, idx in pairs(objRefNames) do
  35.       for k, v in pairs(obj) do
  36.         file:write("multiRefObjects["..idx.."][");
  37.         write(file, k, 0, objRefNames);
  38.         file:write("] = ");
  39.         write(file, v, 0, objRefNames);
  40.         file:write(";\n");
  41.       end;
  42.     end;
  43.     -- Create the remaining objects
  44.     for i = 1, n do
  45.       file:write("local ".."obj"..i.." = ");
  46.       write(file, (select(i,...)), 0, objRefNames);
  47.       file:write("\n");
  48.     end
  49.     -- Return them
  50.     if n > 0 then
  51.       file:write("return obj1");
  52.       for i = 2, n do
  53.         file:write(" ,obj"..i);
  54.       end;
  55.       file:write("\n");
  56.     else
  57.       file:write("return\n");
  58.     end;
  59.     file:close();
  60.   end;

Loading the exported data is simple, but the provided method performs some error checking.

persistence.lua [5.50 kB]

  1.   load = function (path)
  2.     local f, e = loadfile(path);
  3.     if f then
  4.       return f();
  5.     else
  6.       return nil, e;
  7.     end;
  8.   end;

I hope this code is useful for someone, use it as you wish, it is licensed under the MIT license.

Comment [1]

Apple Trailers with MPlayer · 2009-12-18 15:13 by Black in

The trailers on Apple’s Trailer Page are standard Quicktime Movie files, MPlayer can play them without any problem. Unfortunately, Apple seems to have decided to block access to non-quicktime clients. Luckily, the blocking is fairly simple: Clients that send the wrong User Agent are simply ignored.

MPlayer can easily spoof it and display the movie without problems, just add the following to your .mplayer/config file.


  1. [extension.mov]
  2. user-agent="Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540"
  3. cache=10000
  4. cache-min=50

Getting the correct user agent out of MPlayer is fairly easy as well using netcat. Start the program in listening mode and then direct QuickTime Player to the local computer: http://localhost:12345/test.mov

QuickTime HTTP Request Header

  1. > nc -l localhost 12345
  2. GET /test.mov HTTP/1.1
  3. Host: localhost:12345
  4. Range: bytes=0-1
  5. Connection: close
  6. User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
  7. Accept: */*
  8. Accept-Encoding: identity


Creating new user with PowerShell · 2009-12-05 19:35 by Black in

Exchange Server 2007 has removed the ActiveDirectory integration of previous versions, creating a user in AD does no longer also create and link a Mailbox. To create everything properly, the Management Console for Exchange or PowerShell has to be used.

This PowerShell Script creates a new User with parameters set in a GUI. .Net is used to display a Dialog Box, the text boxes are then evaluated and used to create a new user. After that, that user is changed to reflect the remaining settings.

Additional features include the creation of file shares on the server, automatic generation of the E-Mail address with some limited CharSet cleaning, live updating UI, user expiration date setting and more. The whole script is quite customized to the environment it is used in, but I am sure the core can be used by anyone.

An interesting concept this PowerShell Script show are the creation and event based updating of .Net Widgets. updateUI is a function that is called as event handler for the text box, it can execute input validation, update other parts of the UI or do anything else. (See the linked source for more context):

new-user.ps1 [20.72 kB]

  1. $form = new-object System.Windows.Forms.form
  2. $form.Text = "Exchange 2007 User Create Form"
  3. $form.size = new-object System.Drawing.Size(440,550)
  4. $form.AutoSize = $true
  5. $form.AutoSizeMode = "GrowOnly"
  7. ### FirstName
  8. $posY += $lineHeight
  10. # Add FirstName Box
  11. $firstNameTextBox = new-object System.Windows.Forms.TextBox
  12. $firstNameTextBox.Location = new-object System.Drawing.Size($posXControl,$posY)
  13. $firstNameTextBox.size = new-object System.Drawing.Size($controlWidth,$controlHeight)
  14. $firstNameTextBox.add_TextChanged({updateUID})
  15. $form.Controls.Add($firstNameTextBox)

Creating a new User with PowerShell is easy thanks to the new-mailbox cmd-let the Exchange Integration installs. But setting some of the properties was rather complicated. For some, an AD Object has to be generated:

new-user.ps1 [20.72 kB]

  1.     # General Stuff (alternative: use set-user, for some of those)
  2.     $user = get-user -identity $upn
  3.     $aduser = [ADSI]("LDAP://"+$user.DistinguishedName)
  4.     if ($desc -ne "")
  5.     {
  6.       $aduser.description = $jobDescDrop.Text
  7.     }
  8.     if ($phone -ne "")
  9.     {
  10.       $aduser.telephonenumber = $phone
  11.     }
  12.     if ($webpage -ne "")
  13.     {
  14.       $aduser.wwwhomepage = $webpage
  15.     }
  16.     $aduser.company = $company
  17.     $aduser.department = $department
  18.     # FS
  19.     $aduser.profilePath = $pathPro + $alias + "\%osversion%"
  20.     $aduser.homeDrive = "P:"
  21.     $aduser.homeDirectory = $pathBase + $alias + "$"
  22.     # Commit Settings
  23.     $aduser.SetInfo()

Others such as setting the expiration date of an account to “never expires” require to use a more arcane syntax:

new-user.ps1 [20.72 kB]

  1.     # Hard to change Expiration Date is set directly
  2.     #$aduser.psbase.InvokeGet("AccountExpirationDate")
  3.     $aduser.psbase.InvokeSet("AccountExpirationDate", $validUntil)
  4.     $aduser.psbase.CommitChanges()


Encoding Movies with x264 and mplayer · 2009-12-04 17:09 by Black in

An easy way to encode movies is mencoder. Unfortunately, it is fairly outdated and it’s muxers are mostly problematic. A better way is to use the tools directly: mplayer for decoding, x264 to encode the video, a suitable audio encoder, and a muxer for the container format that is desired.

I wrote an encoder script to handle this all easily. The core is using named pipes, constructs that act like files but don’t actually store content, but relay it:

Encoding Core

  1. mkfifo fifo.y4m fifo.wav
  2. x264 fifo.y4m -o out.mkv &
  3. faac -q 128 fifo.wav -o out.m4a &
  4. mplayer in.mpg -vo yuv4mpeg:file=fifo.y4m -ao pcm:file=fifo.wav:fast
  5. mkvmerge -o result.mkv out.mkv out.m4a

What this does: First it creates two named pipes of the desired name. Then the encoder programs for video and audio are started. They read from the named pipes, which blocks until an other process writes data into them. As last process, mplayer is started, writing raw output into the two named pipes with the file based video and audio output module. After encoding is complete, the two resulting video and audio streams are merged. If everything went well, the streams should fit perfectly.

The script also contains a lot of maintenance code that handles encoding and decoding in their own screen sub-sessions. That way the output from all tools can be seen easily… but it’s not really useful… :)

Problems with this method are the inability to handle variable framerate content and depending on the muxer the loss of meta data such as framerate and view aspect ratio/display size.