Go to content Go to navigation

Lua Table Persistence · 2010-01-27 14:56 by Black in

Lua is a very flexible scripting language for embedding into programs. It’s standard API is very slim, it lacks all but basic functions. Adding them is easy though.

The persistence code here requires nothing but lua’s standard io.open for reading and writing files. It can handle loops, multiple references to the same table in both keys and values, and most standard value types.
Not supported are userdata, threads and many types of functions. Exporting simple lua functions works, but the exported byte code is not portable. The result from the export is itself lua code, it can be executed and returns data structures equivalent to those that were exported.

The core for the export is a simple recursion with a dispatcher method and writers for all types. When unsupported types are encountered, nil is written. This can cause problems on import when those unsupported values are used as table keys, but in most cases it is more desirable than to fail the export.

persistence.lua [5.50 kB]

  1. -- Format items for the purpose of restoring
  2. writers = {
  3.   ["nil"] = function (file, item)
  4.       file:write("nil");
  5.     end;
  6.   ["number"] = function (file, item)
  7.       file:write(tostring(item));
  8.     end;
  9.   ["string"] = function (file, item)
  10.       file:write(string.format("%q", item));
  11.     end;
  12.   ["boolean"] = function (file, item)
  13.       if item then
  14.         file:write("true");
  15.       else
  16.         file:write("false");
  17.       end
  18.     end;
  19.   ["table"] = function (file, item, level, objRefNames)
  20.       local refIdx = objRefNames[item];
  21.       if refIdx then
  22.         -- Table with multiple references
  23.         file:write("multiRefObjects["..refIdx.."]");
  24.       else
  25.         -- Single use table
  26.         file:write("{\n");
  27.         for k, v in pairs(item) do
  28.           writeIndent(file, level+1);
  29.           file:write("[");
  30.           write(file, k, level+1, objRefNames);
  31.           file:write("] = ");
  32.           write(file, v, level+1, objRefNames);
  33.           file:write(";\n");
  34.         end
  35.         writeIndent(file, level);
  36.         file:write("}");
  37.       end;
  38.     end;
  39.   ["function"] = function (file, item)
  40.       -- Does only work for "normal" functions, not those
  41.       -- with upvalues or c functions
  42.       local dInfo = debug.getinfo(item, "uS");
  43.       if dInfo.nups > 0 then
  44.         file:write("nil --[[functions with upvalue not supported]]");
  45.       elseif dInfo.what ~= "Lua" then
  46.         file:write("nil --[[non-lua function not supported]]");
  47.       else
  48.         local r, s = pcall(string.dump,item);
  49.         if r then
  50.           file:write(string.format("loadstring(%q)", s));
  51.         else
  52.           file:write("nil --[[function could not be dumped]]");
  53.         end
  54.       end
  55.     end;
  56.   ["thread"] = function (file, item)
  57.       file:write("nil --[[thread]]\n");
  58.     end;
  59.   ["userdata"] = function (file, item)
  60.       file:write("nil --[[userdata]]\n");
  61.     end;
  62. }

To be able to export tables that are referenced several times (be it a cycle in the data structure, or just one that is inserted several times), the structures that are to be written are examined first and the numbers or references to each table are counted.

All tables that have multiple references to them are created at the start in the export file before they are filled with content. This is required, since they could contain themselves or other multi-ref tables.

After all those temporary tables are created, they are filled with content. The writer for tables uses a lookup table for multi-ref tables, instead of creating the table constructor for them, they are assigned from the table created at the start. Last but not least, the passed arguments themselves are created in the same way.

persistence.lua [5.50 kB]

  1.   store = function (path, ...)
  2.     local file, e;
  3.     if type(path) == "string" then
  4.       -- Path, open a file
  5.       file, e = io.open(path, "w");
  6.       if not file then
  7.         return error(e);
  8.       end
  9.     else
  10.       -- Just treat it as file
  11.       file = path;
  12.     end
  13.     local n = select("#", ...);
  14.     -- Count references
  15.     local objRefCount = {}; -- Stores reference that will be exported
  16.     for i = 1, n do
  17.       refCount(objRefCount, (select(i,...)));
  18.     end;
  19.     -- Export Objects with more than one ref and assign name
  20.     -- First, create empty tables for each
  21.     local objRefNames = {};
  22.     local objRefIdx = 0;
  23.     file:write("-- Persistent Data\n");
  24.     file:write("local multiRefObjects = {\n");
  25.     for obj, count in pairs(objRefCount) do
  26.       if count > 1 then
  27.         objRefIdx = objRefIdx + 1;
  28.         objRefNames[obj] = objRefIdx;
  29.         file:write("{};"); -- table objRefIdx
  30.       end;
  31.     end;
  32.     file:write("\n} -- multiRefObjects\n");
  33.     -- Then fill them (this requires all empty multiRefObjects to exist)
  34.     for obj, idx in pairs(objRefNames) do
  35.       for k, v in pairs(obj) do
  36.         file:write("multiRefObjects["..idx.."][");
  37.         write(file, k, 0, objRefNames);
  38.         file:write("] = ");
  39.         write(file, v, 0, objRefNames);
  40.         file:write(";\n");
  41.       end;
  42.     end;
  43.     -- Create the remaining objects
  44.     for i = 1, n do
  45.       file:write("local ".."obj"..i.." = ");
  46.       write(file, (select(i,...)), 0, objRefNames);
  47.       file:write("\n");
  48.     end
  49.     -- Return them
  50.     if n > 0 then
  51.       file:write("return obj1");
  52.       for i = 2, n do
  53.         file:write(" ,obj"..i);
  54.       end;
  55.       file:write("\n");
  56.     else
  57.       file:write("return\n");
  58.     end;
  59.     file:close();
  60.   end;

Loading the exported data is simple, but the provided method performs some error checking.

persistence.lua [5.50 kB]

  1.   load = function (path)
  2.     local f, e = loadfile(path);
  3.     if f then
  4.       return f();
  5.     else
  6.       return nil, e;
  7.     end;
  8.   end;

I hope this code is useful for someone, use it as you wish, it is licensed under the MIT license.

  1. hipe    2010-11-24 00:57    #

    thanks for yr excellent work. i made this of it: git://github.com/hipe/lua-table-persistence.git

    as of release 0.0.2 it’s yr work plus some unit tests, README, etc. I might make it a luarock. i might change its api.

    i hope to make it part of an orbit (mvc) session data solution.

    thx again!

  Textile help