Lua String Writer · 2010-02-04 16:35 by Black in Programming Scripts
Lua strings are opaque byte streams. They are constant, and can only be manipulated by using the string api to create new strings. This can be expensive, especially when creating a string by appending new values at the end. While Lua contains optimizations for direct concatenation, successive appending has a high overhead.
This StringWriter class reduces the overhead by aggregating string concatenations in a table and executing them when requested. It was originally designed to serve as an efficient drop-in replacement for files as created by io.open, but it can also be used standalone.
The class itself is built with a protected shared metatable and state inside a table. The state itself is not protected (it would be possible by using individual metatables or an internal database in a weak table, but this is more elegant). The metatable contains entries to redirect reads to the method table, redirect new writes to nothing and prevent changing or reading the metatable. The concatenation operator is also overloaded, but since it has value semantic, and is not allowed to change the object itself, the implementation is less efficient than StringWriter:write(). Converting a StringWriter with tostring() gives the contained string, equivalent to StringWriter:get().
stringwriter.lua [4.77 kB]
- -- MetaTable for string writers
- local StringWriter_Meta = {
- ["__index"] = StringWriter_Methods;
- ["__newindex"] = function ()
- -- Don't allow setting values
- end;
- ["__metatable"] = StringWriter_ID;
- ["__tostring"] = StringWriter_Methods.get;
- ["__concat"] = function (this, str)
- str = tostring(str);
- local sw = StringWriter();
- sw.string_ = {};
- for _, v in ipairs(this.string_) do
- table.insert(sw.string_, v);
- end
- table.insert(sw.string_, str);
- sw.len_ = this.len_ + #str;
- sw.pos_ = sw.len_;
- return sw;
- end;
- }
The method table itself contains all methods the StringWriter supports. It was modeled after the file class, so many methods are placeholders that do nothing. The methods that are supported are seeking and writing. Seeking simply sets an internal position value. Writing in the context of files means overwriting and extending. When the position is at the end, the contents that are to be written can simply be appended to the contents table. Otherwise, the string has to be baked, split, and recomposed.
stringwriter.lua [4.77 kB]
- -- Methods for string writers
- local StringWriter_Methods = {
- ["close"] = voidFunc;
- ["flush"] = voidFunc;
- ["lines"] = voidFunc;
- ["read"] = voidFunc;
- ["seek"] = function (this, base, offset)
- -- Only act on StringWriters
- if not StringWriter_Check(this) then
- return nil, "Invalid StringWriter";
- end;
- -- Default offset
- if type(base) == "number" then
- offset = base; -- Not done in file, but reasonable
- else
- offset = offset or 0;
- end
- -- Set position and return it
- if base == "set" then
- this.pos_ = math.clamp(offset,0, this.len_);
- elseif base == "end" then
- this.pos_ = math.clamp(#this.string_+offset,0, this.len_);
- else -- "cur"
- this.pos_ = math.clamp(this.pos_+offset,0, this.len_);
- end
- return this.pos_;
- end;
- ["setvbuf"] = voidFunc;
- ["write"] = function (this, ...)
- -- Only act on StringWriters
- if not StringWriter_Check(this) then return end;
- -- Concat all arguments (assuming they are valid)
- local s = table.concat({...});
- -- Concat argument string with current string
- if this.pos_ == -1 or this.pos_ == this.len_ then
- -- Just append
- table.insert(this.string_, s);
- else
- -- Insert, merge into a string
- local sFull = table.concat(this.string_);
- -- Split it up
- local sLeft = string.sub(sFull, 1, this.pos_);
- local sRight = string.sub(sFull, this.pos_+1+#s, -1)
- -- And put it back in
- this.string_ = {sLeft, s, sRight};
- end
- -- Update position
- this.pos_ = this.pos_ + #s;
- if this.pos_ > this.len_ then
- this.len_ = this.pos_;
- end;
- end;
- ["get"] = function (this)
- if not StringWriter_Check(this) then
- return nil, "Invalid StringWriter";
- else
- this.string_ = {table.concat(this.string_)};
- return this.string_[1];
- end;
- end;
- }
StringWriter instances are created by a factory method. It initializes the state and sets the metatable.
stringwriter.lua [4.77 kB]
- -- StringWriter factory
- StringWriter = function ()
- local sw = {
- string_ = {""};
- len_ = 0;
- pos_ = 0;
- }
- setmetatable(sw, StringWriter_Meta);
- return sw;
- end
I hope this code is useful for someone, use it as you wish, it is licensed under the MIT license.
Lua Table Persistence · 2010-01-27 14:56 by Black in Programming Scripts
Lua is a very flexible scripting language for embedding into programs. It’s standard API is very slim, it lacks all but basic functions. Adding them is easy though.
The persistence code here requires nothing but lua’s standard io.open for reading and writing files. It can handle loops, multiple references to the same table in both keys and values, and most standard value types.
Not supported are userdata, threads and many types of functions. Exporting simple lua functions works, but the exported byte code is not portable. The result from the export is itself lua code, it can be executed and returns data structures equivalent to those that were exported.
The core for the export is a simple recursion with a dispatcher method and writers for all types. When unsupported types are encountered, nil is written. This can cause problems on import when those unsupported values are used as table keys, but in most cases it is more desirable than to fail the export.
persistence.lua [5.50 kB]
- -- Format items for the purpose of restoring
- writers = {
- ["nil"] = function (file, item)
- file:write("nil");
- end;
- ["number"] = function (file, item)
- file:write(tostring(item));
- end;
- ["string"] = function (file, item)
- file:write(string.format("%q", item));
- end;
- ["boolean"] = function (file, item)
- if item then
- file:write("true");
- else
- file:write("false");
- end
- end;
- ["table"] = function (file, item, level, objRefNames)
- local refIdx = objRefNames[item];
- if refIdx then
- -- Table with multiple references
- file:write("multiRefObjects["..refIdx.."]");
- else
- -- Single use table
- file:write("{\n");
- for k, v in pairs(item) do
- writeIndent(file, level+1);
- file:write("[");
- write(file, k, level+1, objRefNames);
- file:write("] = ");
- write(file, v, level+1, objRefNames);
- file:write(";\n");
- end
- writeIndent(file, level);
- file:write("}");
- end;
- end;
- ["function"] = function (file, item)
- -- Does only work for "normal" functions, not those
- -- with upvalues or c functions
- local dInfo = debug.getinfo(item, "uS");
- if dInfo.nups > 0 then
- file:write("nil --[[functions with upvalue not supported]]");
- elseif dInfo.what ~= "Lua" then
- file:write("nil --[[non-lua function not supported]]");
- else
- local r, s = pcall(string.dump,item);
- if r then
- file:write(string.format("loadstring(%q)", s));
- else
- file:write("nil --[[function could not be dumped]]");
- end
- end
- end;
- ["thread"] = function (file, item)
- file:write("nil --[[thread]]\n");
- end;
- ["userdata"] = function (file, item)
- file:write("nil --[[userdata]]\n");
- end;
- }
To be able to export tables that are referenced several times (be it a cycle in the data structure, or just one that is inserted several times), the structures that are to be written are examined first and the numbers or references to each table are counted.
All tables that have multiple references to them are created at the start in the export file before they are filled with content. This is required, since they could contain themselves or other multi-ref tables.
After all those temporary tables are created, they are filled with content. The writer for tables uses a lookup table for multi-ref tables, instead of creating the table constructor for them, they are assigned from the table created at the start. Last but not least, the passed arguments themselves are created in the same way.
persistence.lua [5.50 kB]
- store = function (path, ...)
- local file, e;
- if type(path) == "string" then
- -- Path, open a file
- file, e = io.open(path, "w");
- if not file then
- return error(e);
- end
- else
- -- Just treat it as file
- file = path;
- end
- local n = select("#", ...);
- -- Count references
- local objRefCount = {}; -- Stores reference that will be exported
- for i = 1, n do
- refCount(objRefCount, (select(i,...)));
- end;
- -- Export Objects with more than one ref and assign name
- -- First, create empty tables for each
- local objRefNames = {};
- local objRefIdx = 0;
- file:write("-- Persistent Data\n");
- file:write("local multiRefObjects = {\n");
- for obj, count in pairs(objRefCount) do
- if count > 1 then
- objRefIdx = objRefIdx + 1;
- objRefNames[obj] = objRefIdx;
- file:write("{};"); -- table objRefIdx
- end;
- end;
- file:write("\n} -- multiRefObjects\n");
- -- Then fill them (this requires all empty multiRefObjects to exist)
- for obj, idx in pairs(objRefNames) do
- for k, v in pairs(obj) do
- file:write("multiRefObjects["..idx.."][");
- write(file, k, 0, objRefNames);
- file:write("] = ");
- write(file, v, 0, objRefNames);
- file:write(";\n");
- end;
- end;
- -- Create the remaining objects
- for i = 1, n do
- file:write("local ".."obj"..i.." = ");
- write(file, (select(i,...)), 0, objRefNames);
- file:write("\n");
- end
- -- Return them
- if n > 0 then
- file:write("return obj1");
- for i = 2, n do
- file:write(" ,obj"..i);
- end;
- file:write("\n");
- else
- file:write("return\n");
- end;
- file:close();
- end;
Loading the exported data is simple, but the provided method performs some error checking.
persistence.lua [5.50 kB]
- load = function (path)
- local f, e = loadfile(path);
- if f then
- return f();
- else
- return nil, e;
- end;
- end;
I hope this code is useful for someone, use it as you wish, it is licensed under the MIT license.
Comment [1]
Apple Trailers with MPlayer · 2009-12-18 15:13 by Black in Scripts
The trailers on Apple’s Trailer Page are standard Quicktime Movie files, MPlayer can play them without any problem. Unfortunately, Apple seems to have decided to block access to non-quicktime clients. Luckily, the blocking is fairly simple: Clients that send the wrong User Agent are simply ignored.
MPlayer can easily spoof it and display the movie without problems, just add the following to your .mplayer/config
file.
.mplayer/config
- [extension.mov]
- user-agent="Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540"
- cache=10000
- cache-min=50
Getting the correct user agent out of MPlayer is fairly easy as well using netcat. Start the program in listening mode and then direct QuickTime Player to the local computer: http://localhost:12345/test.mov
QuickTime HTTP Request Header
- > nc -l localhost 12345
- GET /test.mov HTTP/1.1
- Host: localhost:12345
- Range: bytes=0-1
- Connection: close
- User-Agent: Apple Mac OS X v10.6.2 CoreMedia v1.0.0.10C540
- Accept: */*
- Accept-Encoding: identity
Creating new user with PowerShell · 2009-12-05 19:35 by Black in Scripts
Exchange Server 2007 has removed the ActiveDirectory integration of previous versions, creating a user in AD does no longer also create and link a Mailbox. To create everything properly, the Management Console for Exchange or PowerShell has to be used.
This PowerShell Script creates a new User with parameters set in a GUI. .Net is used to display a Dialog Box, the text boxes are then evaluated and used to create a new user. After that, that user is changed to reflect the remaining settings.
Additional features include the creation of file shares on the server, automatic generation of the E-Mail address with some limited CharSet cleaning, live updating UI, user expiration date setting and more. The whole script is quite customized to the environment it is used in, but I am sure the core can be used by anyone.
An interesting concept this PowerShell Script show are the creation and event based updating of .Net Widgets. updateUI is a function that is called as event handler for the text box, it can execute input validation, update other parts of the UI or do anything else. (See the linked source for more context):
new-user.ps1 [20.72 kB]
- $form = new-object System.Windows.Forms.form
- $form.Text = "Exchange 2007 User Create Form"
- $form.size = new-object System.Drawing.Size(440,550)
- $form.AutoSize = $true
- $form.AutoSizeMode = "GrowOnly"
- ### FirstName
- $posY += $lineHeight
- # Add FirstName Box
- $firstNameTextBox = new-object System.Windows.Forms.TextBox
- $firstNameTextBox.Location = new-object System.Drawing.Size($posXControl,$posY)
- $firstNameTextBox.size = new-object System.Drawing.Size($controlWidth,$controlHeight)
- $firstNameTextBox.add_TextChanged({updateUID})
- $form.Controls.Add($firstNameTextBox)
Creating a new User with PowerShell is easy thanks to the new-mailbox cmd-let the Exchange Integration installs. But setting some of the properties was rather complicated. For some, an AD Object has to be generated:
new-user.ps1 [20.72 kB]
- # General Stuff (alternative: use set-user, for some of those)
- $user = get-user -identity $upn
- $aduser = [ADSI]("LDAP://"+$user.DistinguishedName)
- if ($desc -ne "")
- {
- $aduser.description = $jobDescDrop.Text
- }
- if ($phone -ne "")
- {
- $aduser.telephonenumber = $phone
- }
- if ($webpage -ne "")
- {
- $aduser.wwwhomepage = $webpage
- }
- $aduser.company = $company
- $aduser.department = $department
- # FS
- $aduser.profilePath = $pathPro + $alias + "\%osversion%"
- $aduser.homeDrive = "P:"
- $aduser.homeDirectory = $pathBase + $alias + "$"
- # Commit Settings
- $aduser.SetInfo()
Others such as setting the expiration date of an account to “never expires” require to use a more arcane syntax:
new-user.ps1 [20.72 kB]
- # Hard to change Expiration Date is set directly
- #$aduser.psbase.InvokeGet("AccountExpirationDate")
- $aduser.psbase.InvokeSet("AccountExpirationDate", $validUntil)
- $aduser.psbase.CommitChanges()
Encoding Movies with x264 and mplayer · 2009-12-04 17:09 by Black in Scripts
An easy way to encode movies is mencoder. Unfortunately, it is fairly outdated and it’s muxers are mostly problematic. A better way is to use the tools directly: mplayer for decoding, x264 to encode the video, a suitable audio encoder, and a muxer for the container format that is desired.
I wrote an encoder script to handle this all easily. The core is using named pipes, constructs that act like files but don’t actually store content, but relay it:
Encoding Core
- mkfifo fifo.y4m fifo.wav
- x264 fifo.y4m -o out.mkv &
- faac -q 128 fifo.wav -o out.m4a &
- mplayer in.mpg -vo yuv4mpeg:file=fifo.y4m -ao pcm:file=fifo.wav:fast
- mkvmerge -o result.mkv out.mkv out.m4a
What this does: First it creates two named pipes of the desired name. Then the encoder programs for video and audio are started. They read from the named pipes, which blocks until an other process writes data into them. As last process, mplayer is started, writing raw output into the two named pipes with the file based video and audio output module. After encoding is complete, the two resulting video and audio streams are merged. If everything went well, the streams should fit perfectly.
The script also contains a lot of maintenance code that handles encoding and decoding in their own screen sub-sessions. That way the output from all tools can be seen easily… but it’s not really useful… :)
Problems with this method are the inability to handle variable framerate content and depending on the muxer the loss of meta data such as framerate and view aspect ratio/display size.