Handling constants as record types attributes

Then one day i discovered that our JAVA team implements constants as a simple SQL function that returns one record where the statuses textual code is a out parameter and the integer value of the status is kept inside that parameter.

CREATE OR REPLACE FUNCTION order_statuses(

    OUT CREATED int,

    OUT DELIVERED int,

    OUT CANCELLED int

) RETURNS record AS

$$

BEGIN

     CREATED := 1;

     DELIVERED := 2;

     CANCELLED := 3;

END;

$$ LANGUAGE plpgsql;

This is a really neat idea, you only have to select the output of the function into a record type variable and Voilà!

C_ORDER = order_statuses();

SELECT * FROM orders WHERE key_user = i_username AND key_status = C_ORDER.DELIVERED;

Note that this kind of approach also has it’s drawbacks: The query is unable to use partial indexes in the plan and when you are scanning a table solely based on this key then hardcoding a constant will give you a more correct plan as it is able to use the statistics collected for the column.

No need to declare anything besides the C_ORDER record and code is readable, no integer constants inside.
However such an approach has it’s shortcomings, when you add a new status you need to drop the previous constant returning function and create a new one, this is quite annoying when you have all the same statuses stored inside a table and could fetch them from there. However the limitations that a function must always return a predefined fixed count of output variables will not enable you to return a record with dynamical length.

creating records with variable length

Then i figured out that this approach actually works if we do this one little exception to our “no SQL should be dynamically created inside DB” policy. It’s perfectly ok to create dynamic SQL if you are not doing any data access within it – just for converting data. (Doing data fetching inside dynamically generated SQL makes monitoring the database performance and access patterns a pain in the ass).

So here is what i came up with:

CREATE TABLE classificator.classificator(

    id_classificator serial PRIMARY KEY,

    classificator text,

    code int,

    lookup text,

    description text

);

CREATE INDEX idx_classificator ON classificator.classificator(classificator);INSERT INTO classificator.classificator (classificator, code, lookup, description) VALUES ('order_status',1,'CREATED','Order created');

INSERT INTO classificator.classificator (classificator, code, lookup, description) VALUES ('order_status',2,'DELIVERED','Order delivered');

INSERT INTO classificator.classificator (classificator, code, lookup, description) VALUES ('order_status',3,'CANCELLED','Order cancelled');

CREATE AGGREGATE array_accum (anyelement)

(

sfunc = array_append,

stype = anyarray,

initcond = '{}'

);

CREATE OR REPLACE FUNCTION classificator.get_constant_structure_sql(

    IN i_classificator text,

    OUT exec_str text

) RETURNS text AS $$

DECLARE

    _lookup text[];

    _key_integer text[];

    _keys text;

    _vals text;

BEGIN

    SELECT array_accum('"'||c.lookup||'"'),

           array_accum(c.code)

      FROM classificator.classificator c

    WHERE c.classificator = i_classificator

      INTO _lookup,

           _key_integer;

    IF NOT FOUND THEN

         RETURN;

    END IF;

_keys = array_to_string(_lookup, ', ');

    _vals = array_to_string(_key_integer, ', ');

    exec_str = 'SELECT * FROM (VALUES ('||_vals||')) AS t ('||_keys||')';

    RETURN;

END

$$ LANGUAGE plpgsql SECURITY DEFINER;

As you can see i don’t care much about the 3NF (Third Normalized Form) standards to keep all status codes inside separate tables. I personally am not a big fan of 3NF as it has it’s own drawbacks. Having a swarm of small status codes & other similar tables in your system makes it too difficult to locate anything. However if you like you can create this function in a way that it handles only one table (eg. it creates the status codes record only for the table order_statuses).

The implementation into a function is really simple as you can see from the following example:

...

DECLARE

    _rec_struct text;

    C_ORDER record;

BEGIN

    _rec_struct = classificator.get_constant_structure_sql('order_status');

    EXECUTE _rec_struct INTO C_ORDER;

    SELECT * FROM orders WHERE key_status = C_ORDER."DELIVERED";

...

Code is clean, simple, readable and consumes only 3 rows for every constant set that you need defined. Works also nicely as a eval check so you can not implement constants in your code that are not yet in the table.

Here’s a small data dump to visualize whats going on inside these functions:

select * from classificator.classificator;

 id_classificator | classificator | code |  lookup   |   description

------------------+---------------+------+-----------+-----------------

                1 | order_status  |    1 | CREATED   | Order created

                2 | order_status  |    2 | DELIVERED | Order delivered

                3 | order_status  |    3 | CANCELLED | Order cancelled

(3 rows)select * from classificator.get_constant_structure_sql('order_status');

                                  exec_str

-----------------------------------------------------------------------------

 SELECT * FROM (VALUES (3, 2, 1)) AS t ("CANCELLED", "DELIVERED", "CREATED")

(1 row)

SELECT * FROM (VALUES (3, 2, 1)) AS t ("CANCELLED", "DELIVERED", "CREATED");

 CANCELLED | DELIVERED | CREATED

-----------+-----------+---------

         3 |         2 |       1

(1 row)

So now we have the possibility to create a record inside every Pl/PgSQL function that holds all the constants, changes dynamically on adding new constants and spares us a lot of code in function DECLARE parts. Hope this idea was useful, i for sure plan to implement it wherever possible

Promo corner

is looking for a DB QA engineer, the job is located in Tallinn. The team is small and brilliant, this is a great opportunity if you are a quick learner and are interested how big database clusters actually work!

    Table "public.tmp_email"
  Column  |   Type   | Modifiers
----------+----------+-----------
 email    | text     |
 trigrams | tsvector |

trigrams is a column added solely for the purpose of being able to help us to search emails by substrings, it contains all the unique sequential 3 letter combinations used in the email field. For example for an email ‘kristo@eesti.ee’ it would consist of

o@e est sti .ee @ee sto ris ees ist ti. kri i.e to@

We will do this with a simple plpython function (can be done in plpgsql also but it looks nicer in python)

CREATE FUNCTION text_to_trigrams(i_txt text) RETURNS text AS $$
    trigrams = {}
    for i in range(len(i_txt)-2):
        trigrams[i_txt[i:i+3]] = 1
    return ' '.join(trigrams.keys())
$$ LANGUAGE plpythonu SECURITY DEFINER;

In a production system you would probably be calculating the trigrams in an update/insert trigger on the table doing the same operation provided in the following query:

update tmp_email set trigrams = to_tsvector('simple',text_to_trigrams(email));

The principle itself is really simple, after splitting up the string in email column into these unique 3 letter combinations we will create a full text index on that column (trigram column) which will enable us to search for a predefined set of trigrams inside a string.

create index idx_email_substring on tmp_email using gin (trigrams);

Be warned that the index is quite hefty. In my testcase with live e-mail addresses the index was just a bit bigger than the table itself, remember i warned you about the drawbacks However the search itself has a quite acceptable performance, but before doing the search we will have to create the search condition. This is achived by creating a query that consists of the trigrams that we are looking for, eg. we want to find users with e-mail address that includes the substring ‘%to@skype%’

test=# select replace(text_to_trigrams('to@skype'),' ',' & ');
              replace
-----------------------------------
 o@s & sky & @sk & ype & to@ & kyp

This query will find all the trigrams that are contained within the string and replace the spaces we needed for tsvector creation with & operations needed for the tsquery. So using this generated string we can do the search in the following way:

test=# explain analyze select * from tmp_email where trigrams @@ to_tsquery('simple','o@s & sky & @sk & ype & to@ & kyp');
                                                            QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------
 Index Scan using idx_email_substring on tmp_email  (cost=0.00..15.50 rows=14 width=64) (actual time=0.086..3.349 rows=2 loops=1)
   Index Cond: (trigrams @@ '''o'' & ''s'' & ''sky'' & ''sk'' & ''ype'' & ''to'' & ''kyp'''::tsquery)
 Total runtime: 3.379 ms

test=# explain analyze select count(*) from tmp_email ;
                                                     QUERY PLAN
--------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=676.60..676.61 rows=1 width=0) (actual time=45.656..45.657 rows=1 loops=1)
   ->  Seq Scan on tmp_email  (cost=0.00..642.28 rows=13728 width=0) (actual time=0.009..27.038 rows=13728 loops=1)
 Total runtiows=13728 loops=1) Total runtime: 45.701 ms(3 rows)test=# select count(*) from tmp_email ; count ------- 13728

test=# select count(*) from tmp_email ;
 count
-------
 13728

I know that this is not the nicest way to do this but until we have some generic substring search implemented into PostgreSQL this is at least one way to do it for certain cases. If you want to know in more detail how the previously described solution works i suggest you take a loot at the GIN spec. Be aware that this currently only works if the substring is at least 3 characters long.

test=# begin;
BEGIN
test=# select now();
              now
-------------------------------
 2007-11-02 16:57:38.011621+02
(1 row)

test=# select now();
              now
-------------------------------
 2007-11-02 16:57:38.011621+02
(1 row)

test=# commit;
COMMIT

The time returned is always the transaction start date (don’t remember exactly though if it was the time of the begin statement or first SQL statement). However this can be overcome, there is a function that for whatever reasons i don’t know/remember and also really don’t care acts differently – it’s called timeofday().

test=# begin;
BEGIN
test=# select timeofday()::timestamptz;
           timeofday
-------------------------------
 2007-11-02 17:01:15.042648+02
(1 row)

test=# select timeofday()::timestamptz;
           timeofday
-------------------------------
 2007-11-02 17:01:19.266846+02
(1 row)

test=# commit;
COMMIT

Thanks to Ze / Hannu for pointing this out to me (bow)

In this post we will look at the skytools scripting framework in general and also look into writing simple queue consumers. There are quite a lot of tasks in the database that don’t need immediate completion and can aswell run in the background. A simple example for this would be sending out e-mails of user creation / password reminders etc. Usually these kind of batch jobs are done by using a queue table that the script fetches data from and after completion removes the row from table. PgQ enables us to do this even more conveniently and is a lot more effective performancewise.

PgQ is the queueing solution that empowers the londiste replication. When you have londiste installed in your database you have also PgQ installed.
So first of all let’s bring out a few key points why it’s better to use PgQ instead of queue tables:

• Performance : cleanup is done by rotating between 3 tables and using truncate to get rid of old data, no need for delete queries
• Scalability : one PgQ queue can have basically unlimited consumers that keep their own high watermark and share the data
• Retry queue: if a queue message can not be processed instantly it can be moved to the retry queue that will automatically reinsert the events into main queue later

We will not look at any of above topics in this post but i promise i will do this sometime in the future…

setup

If you already have londiste installed you don’t have to do anything.
If not locate the txid.sql & pgq.sql file inside the skytools frame and load it into the database in the same order
Create configuration file for the ticker and put the ticker daemon running. If you don’t know what i’m talking about look at the ticker setup section in clustering with plproxy part II
Create the queue

select * from pgq.create_queue('mailer');

This should generate the result of ’1′ indicating that Markos brain was in set_no_documentation = 1 mode while writing code
More informative would be to do a select on the pgq.queue or in older versions pgq.queue_config table:

queries=# select * from pgq.queue;
-[ RECORD 1 ]------------+------------------------------
queue_id | 1
queue_name | mailer
queue_ntables | 3
queue_cur_table | 0
queue_rotation_period | 02:00:00
queue_switch_step1 | 1057688
queue_switch_step2 | 1057688
queue_switch_time | 2007-10-19 18:57:55.775194+03
queue_external_ticker | f
queue_ticker_max_count | 500
queue_ticker_max_lag | 00:00:03
queue_ticker_idle_period | 00:01:00
queue_data_pfx | pgq.event_1
queue_event_seq | pgq.event_1_id_seq
queue_tick_seq | pgq.event_1_tick_seq


A peek inside the queue

The queue data itself is in 3+1 tables the pgq.event_X table is the table that the other 3 inherit:

queries=# \dt pgq.event_
pgq.event_1 pgq.event_1_0 pgq.event_1_1 pgq.event_1_2 pgq.event_template

So if you want to look at the data inside the queue the main table is enough.
The event itself is simple it consists of 2 fields that are filled by the user ev_type & ev_data both of which are text fields.
An example of ev_type field values could be ‘I’,'U’,'D’ for replication actions (Insert, Update, Delete)
Ev data contains all the data that you want to send, it can be a single value but we usually go for urlencoded strings:

"user=kristokaiv&email=kristo.kaiv@geemail.com"

as the tools in skytools framework support this format.

Putting messages to queue

For inserting events we ourselves usually use the following trick:
create a dummy table that is used only to define the events structure

queries=# create schema queue;
CREATE SCHEMA
queries=# create table queue.welcome_email(username text, language text, firstname text, lastname text);
CREATE TABLE

add a pgq.logutriga to the table, what pgq.logutriga does is it urlencodes the inserted column field pairs: column1=value1&column2=value2..
and inserts it to the queue given as the parameter. The other parameter is either ‘SKIP’ or ‘OK’. Skip meaning that data is discarded after trigger has processed it. ‘OK’ meaning that data will be actually inserted to the table. Both of them are useful but mostly you don’t actually need the data for any other reasons than just debugging.

queries=# CREATE TRIGGER ins_to_queue BEFORE INSERT ON queue.welcome_email FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('mailer', 'welcome_email', 'SKIP');
CREATE TRIGGER

or if you don’t have the latest PgQ version then:

queries=# CREATE TRIGGER ins_to_queue BEFORE INSERT ON queue.welcome_email FOR EACH ROW EXECUTE PROCEDURE pgq.logutriga('mailer','SKIP');
CREATE TRIGGER

So finally we come to the actual event creation:

queries=# insert into queue.welcome_email (username, language, firstname, lastname) values ('kristokaiv','Kristo','Kaiv','kristo.kaiv@geemail.com');
INSERT 0 0
queries=# select * from pgq.event_1 where ev_data like '%kristokaiv%';
-[ RECORD 1 ]------------------------------------------------------------------------------------
ev_id | 2
ev_time | 2007-10-19 19:56:53.353656+03
ev_txid | 1057748
ev_owner |
ev_retry |
ev_type | I:
ev_data | username=kristokaiv&language=Kristo&firstname=Kaiv&lastname=kristo.kaiv%40geemail.com
ev_extra1 | queue.welcome_email
ev_extra2 |
ev_extra3 |
ev_extra4 |

As you can see the for every column in our dummy table we have the inserted value and everything is nicely encoded. Also the name of the table on which the trigger resides is added to one of the extra columns. This enables us to use multiple dummy queue tables to insert to one queue table.

If you don’t like this solution much you could or would like to use different encoding you are absolutely free to do this. When dummy tables are not your favourite weapon of choice but urlencoding is fine then you could for example create the following function:

CREATE OR REPLACE FUNCTION public.urlencode (text, text)  RETURNS text AS $$
    import skytools
    key_value = {args[0]:args[1]}
    return skytools.db_urlencode (key_value)
$$ LANGUAGE plpythonu VOLATILE SECURITY DEFINER;

And later on explicitly add the key, value pairs together:

perform pgq.insert_event('mailer','welcome_email',urlencode('username','kristo.kaiv')||'&'||urlencode('email','kristo.kaiv@geemail.com'));

but this dummy table based solution is imho quite easy to maintain.

consuming the event

This is what our consumer looks like:

import sys, os, pgq, skytools

class Mailer(pgq.Consumer):
    def sendWelcomeMail(self, params):
        """try to send mail, return true on success, false on failure"""
        return True

    def process_batch(self, src_db, batch_id, ev_list):
        for ev in ev_list:
            d = skytools.db_urldecode(ev.data)
            self.log.debug ("event : %s | type : %s | inserted by : %s" % (d, ev.type, ev.extra1))
            if not self.sendWelcomeMail(d):
                sys.exit(1)
            ev.tag_done()

if __name__ == '__main__':
    script = Mailer("mailer_daemon","src_db",sys.argv[1:])
    script.start()

Quite short isn’t it?
I didn’t actually remove the mail sending part but it was never there. I admit i have absolutely no idea how to send an e-mail from python. Sending them however isn’t probably also the topic you are interested in, so let’s skip it.
The parameters that are given to the pgq.Consumer are:

• name of the configuration section in config file : “mailer_daemon”
• name of parameter from config file that contains the connection string to PgQ database : “src_db”
• command line arguments

Our configuration file looks like this

[mailer_daemon]
job_name = mailer_daemon
src_db = dbname=queries
pgq_queue_name = mailer
logfile = %(job_name)s.log
pidfile = %(job_name)s.pid


running the script & monitoring

Start the script with:

python mailer.py mailer.conf -v

-v means all self.log.debug info will be also displayed, this is useful for debugging what your script does. To run the script as a background process use the key -d. If you start the process in background be sure you have a pid file defined in the script config.
The script will start producing output that looks like this:

mbpro:~/temp kristokaiv$ python mailer.py mailer.conf -v
2007-10-19 20:59:59,968 7634 DEBUG Attaching
2007-10-19 21:00:00,002 7634 DEBUG event : {'hi!': None} | type : 4 | inserted by : None
2007-10-19 21:00:00,002 7634 DEBUG event : {'username': 'kristokaiv', 'lastname': 'kristo.kaiv@geemail.com', 'firstname': 'Kaiv', 'language': 'Kristo'} | type : I: | inserted by : queue.welcome_email
2007-10-19 21:00:00,011 7634 INFO {count: 2, duration: 0.0260310173035}
2007-10-19 21:00:00,018 7634 INFO {count: 0, duration: 0.00694489479065}

You can monitor the queue status in the following way:

mbpro:~/temp kristokaiv$ pgqadm.py ticker2.ini status
Postgres version: 8.2.3   PgQ version: 2.1.5

Event queue                                    Rotation        Ticker   TLag
------------------------------------------------------------------------------
mailer                                          3/7200s    500/3s/60s     2s

If you need to automate reporting the queue statuses you can find it by calling

select * from pgq.get_queue_info();

in the database. This concludes the short introduction to writing queue based batch jobs using PgQ and skytools framework

The simplest real world usage for plproxy would be it’s use for redirecting read-only queries to read-only replicas of master database. The replicated databases can be filled with data via Londiste that is part of the SkyTools package, setup tutorial can be found here or with Slony which is a more heavyweight solution but from my own experience also harder to setup and maintain though definitely at the time being better documented.
A typical read-only cluster could look like on the following schema. The databases with the letter (P) on them are connection poolers. We ourself use PgBouncer but pgpool is also a choice.
The poolers are needed to minimize the number of open connections to a database also execution plans are cached on a connection basis. Of course everything will work fine also without the poolers. Dashed bold arrows represent replicas.

In this setup the plproxy functions determine the database to which the query is redirected. Read&write queries go to master database and read-only queries are distributed based on the algorithm you define to read-only replicas.
Setting up replication itself is relatively easy once you have passed the painful skytools installation process.
First let us create both replicas from write database toward ro1 & ro2. ro1 configuration file looks like this:
replica1.ini

[londiste]
job_name = londiste_master_to_r1
provider_db = dbname=write
subscriber_db = dbname=ro1
# it will be used as sql ident so no dots/spaces
pgq_queue_name = londiste.write
pidfile = %(job_name)s.pid
logfile = %(job_name)s.log
use_skylog = 0

replica2.ini is basically the same only job name and database name need to be changed. Now let’s install Londiste on provider (write) and subscribers (ro1,ro2) and start the replication daemons:

mbpro:~/temp kristokaiv$ londiste.py replica1.ini provider install
mbpro:~/temp kristokaiv$ londiste.py replica1.ini subscriber install
mbpro:~/temp kristokaiv$ londiste.py replica2.ini subscriber install
mbpro:~/temp kristokaiv$ londiste.py replica1.ini replay -d
mbpro:~/temp kristokaiv$ londiste.py replica2.ini replay -d

The next thing you need to do is to setup the ticker process on the database where write is performed. The ticker creates sync events so running it with shorter intervals will reduce latency. My configuration file looks like this:
ticker_write.ini

[pgqadm]
job_name = ticker_write
db = dbname=write
# how often to run maintenance [minutes]
maint_delay_min = 1
# how often to check for activity [secs]
loop_delay = 0.1
logfile = %(job_name)s.log
pidfile = %(job_name)s.pid
use_skylog = 0

To start the ticker as a daemon just run:

mbpro:~/temp kristokaiv$ pgqadm.py ticker_write.ini ticker -d

Lets create a simple table that we will replicate from master to read-only’s

mbpro:~/temp kristokaiv$ psql -c "CREATE TABLE users (username text primary key, password text);" write
mbpro:~/temp kristokaiv$ psql -c "CREATE TABLE users (username text primary key, password text);" ro1
mbpro:~/temp kristokaiv$ psql -c "CREATE TABLE users (username text primary key, password text);" ro2

And add it to replication

mbpro:~/temp kristokaiv$ londiste.py replica1.ini provider add users
mbpro:~/temp kristokaiv$ londiste.py replica1.ini subscriber add users
mbpro:~/temp kristokaiv$ londiste.py replica2.ini subscriber add users

After some time the tables should be up to date. Insert a new record in the write database and check if it’s delivered to both read-only db’s.
The functions to insert and select from users table:

CREATE OR REPLACE FUNCTION public.add_user(
    in i_username text,
    in i_password text,
    out status_code text
) AS $$
BEGIN
    PERFORM 1 FROM users WHERE username = i_username;
    IF NOT FOUND THEN
        INSERT INTO users (username, password) VALUES (i_username, i_password);
        status_code = 'OK';
    ELSE
        status_code = 'user exists';
    END IF;
    RETURN;
END; $$ LANGUAGE plpgsql SECURITY DEFINER;

GRANT EXECUTE ON FUNCTION public.add_user(
    in i_username text,
    in i_password text,
    out status_code text
) TO plproxy;

CREATE OR REPLACE FUNCTION login(
    in i_username text,
    in i_password text,
    out status_code text
) AS $$
BEGIN
    SELECT 'OK' FROM users u WHERE username = i_username AND password = i_password INTO status_code;
    IF NOT FOUND THEN status_code = 'FAILED'; END IF;
    RETURN;
END; $$ LANGUAGE plpgsql SECURITY DEFINER;

GRANT EXECUTE ON FUNCTION login(
    in i_username text,
    in i_password text,
    out status_code text
) TO plproxy;

Just for the comfort of those actually trying to follow these steps, here is how the proxy databases
cluster config:

CREATE OR REPLACE FUNCTION plproxy.get_cluster_partitions (cluster_name text)
RETURNS SETOF text AS $$
BEGIN
   IF cluster_name = 'readonly' THEN
        RETURN NEXT 'host=127.0.0.1 dbname=ro1';
        RETURN NEXT 'host=127.0.0.1 dbname=ro2';
        RETURN;
    ELSIF cluster_name = 'write' THEN
        RETURN NEXT 'host=127.0.0.1 dbname=write';
        RETURN;
    END IF;
    RAISE EXCEPTION 'no such cluster%', cluster_name;
END; $$ LANGUAGE plpgsql SECURITY DEFINER;

CREATE OR REPLACE FUNCTION plproxy.get_cluster_config(
    in cluster_name text,
    out key text,
    out val text)
RETURNS SETOF record AS $$
BEGIN
    RETURN;
END;
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION plproxy.get_cluster_version(cluster_name text) RETURNS int AS $$
    SELECT 1;
$$ LANGUAGE SQL;

The last thing left to do is to create the plproxy function definitions that will redirect the login function calls against read-only databases and add_user calls against write database:

CREATE OR REPLACE FUNCTION public.login(
    in i_username text,
    in i_password text,
    out status_code text
) AS $$
CLUSTER 'readonly'; RUN ON ANY;
$$ LANGUAGE plproxy;

CREATE OR REPLACE FUNCTION public.add_user(
    in i_username text,
    in i_password text,
    out status_code text
) AS $$
CLUSTER 'write';
$$ LANGUAGE plproxy;

This is it, the read-only cluster is ready. Note that even though creating such a read-only cluster seems simple and a quick solution for your performance problems it is not a silver bullet solution. Asynchronous replication often creates more problems than it solves so be careful to replicate only non-timecritical data or guarantee a fallback solution when data is not found (eg. proxy function first checks readonly database and if data is not found looks the data up from write db)

Partitioning for dummys

Partitioning let’s you distribute the database load and data between multiple database servers.
The principle itself is simple. Let’s say you have one table that contains the users login credentials but the problem is that there are millions of users that daily log in to their account. This of cause creates a lot of load not talking about the huge table it needs for storage. First we need a criteria based on which we choose what server contains what data. We could do this by the usernames first character. Users from a-j go to first server and usernames beginning with k-z go to the second server. It does work but probably one of the servers gets more load than the other. The most common option is to choose the partition based on the hash of the primary key value, in our case the username. Using a hashing function will distribute the users between servers very evenly. What you need to know about hashing is that hash function basically calculates a number based on any given input it can handle:

select hashtext('kristokaiv1') = 1116512480
select hashtext('kristokaiv2') = 1440348351
select hashtext('kristokaiv3') = -219299073

How it works internally is beyond the scope of this post.
Now let’s say we have 2 partitions, then we could get the partition number based on the username hash like this

partition nr = hashtext($1) & 1

The & 1 will give us the last bit of the number which can be either 0 or 1 which is the number we will use to choose the partition the user data will be stored in. If it’s 0 the data goes to partition 0 and if it’s one the data goes to partition 1

select hashtext('kristokaiv1') & 1 = 0 -> partition 0
select hashtext('kristokaiv2') & 1 = 1 -> partition 1
select hashtext('kristokaiv3') & 1 = 1 -> partition 1

How plproxy works

The concept itself is rather simple – plproxy is a new language created inside the PostgreSQL database that enables to make remote database calls exactly as you do with dblink. The syntax is really straightforward – the following statement creates a new plproxy function in the database that when run will connect to the database remotedb, execute the function get_user_email(text) and return the results.

localdb=#
CREATE FUNCTION get_user_email(username text) RETURNS text AS $$
    CONNECT 'dbname=remotedb host=123.456.123.21 user=myuser';
$$ LANGUAGE plproxy;

Lets create a dummy function in the remotedb that will respond to the call

remotedb=#
create function get_user_email(text)
returns text as $$
    select 'me@somewhere.com'::text;
$$ language sql;

On execution we will see exactly the same results as we would when executing the query on remotedb

localdb=#
select * from get_user_email('tere');
get_user_email
------------------
me@somewhere.com
(1 row)

Of course this is just a really simple example and i will get back to the more complex syntax later, let’s first take a look on how to install the plproxy language.

Installing plproxy

Plproxy can be downloaded from http://pgfoundry.org/projects/plproxy/ but i strongly suggest you get the newest version from the pgfoundry CVS, instructions how to set it up are here. You have to have the PostgreSQL developement environment installed and the folder where the PostgreSQL configuration info tool (pg_config) is needs to be in included in your $PATH variable. If those prerequisites are met then the installation is simple:

$ make
$ make install
$ make installcheck

If you don’t manage to get it working by yourself there is always the mailing list to help you get started.
The final step is to install the language into the database, this can’t be done as for other languages (createlang utility) instead you have to execute the plproxy.sql file that will create the language call handler. locate plproxy.sql is how i found it but it should be somewhere under contrib.

$ psql -f /usr/local/pgsql/share/contrib/plproxy.sql queries
CREATE FUNCTION
CREATE LANGUAGE

Now everything should be done and you can test the setup with the simple plproxy function in the syntax example.

Setting up our first cluster

Let’s create a simple cluster that consists of 3 databases (in my example they are all running on the same PostgreSQL instance). One proxy database called queries and 2 partitions queries_0000 and queries_0001. Horizontal partitioning is done based on username, It’s the most common way for partitioning as most of the data in the database is usually user related eg. users login, users orders, users payments, users settings…

The database cluster setup is stored inside plpgsql functions that plproxy calls.
There are 3 functions that you _MUST_ create for the cluster configuration to work properly. So let’s create them on the proxy database “queries”.

1) plproxy.get_cluster_version(cluster_name text)
This function called on each request and is used to determine if the configuration for a cluster has been changed, if the version number it returns is higher than the cached version number partitions configuration is reloaded. Let’s start with the first version of our configuration like this:

CREATE OR REPLACE FUNCTION plproxy.get_cluster_version(cluster_name text) RETURNS int AS $$
BEGIN
    IF cluster_name = 'queries' THEN
        RETURN 1;
    END IF;
END;
$$ LANGUAGE plpgsql;

2) plproxy.get_cluster_partitions(cluster_name text)
Function should return the connection strings for all partitions in the correct order.
Because of some unreasonable limitation the total count must be power of 2. This is a unreasonable limitation that can easily be overcome but let’s discuss this in another post.

CREATE OR REPLACE FUNCTION plproxy.get_cluster_partitions(cluster_name text) RETURNS SETOF text AS $$
BEGIN
    IF cluster_name = 'queries' THEN
        RETURN NEXT 'host=127.0.0.1 dbname=queries_0000';
        RETURN NEXT 'host=127.0.0.1 dbname=queries_0001';
        RETURN;
    END IF;
    RAISE EXCEPTION 'no such cluster: %', cluster_name;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;

If postgres username is not specified in the connection string the name of CURRENT_USER will be used. As plproxy does not know any passwords, the partition databases should trust connections from the proxy database.

3) plproxy.get_cluster_config(cluster_name text)
This is the equivalent of an init file. It should return the configuration parameters as key – value pairs. All of them are optional but you still need the dummy placeholder function:

CREATE OR REPLACE FUNCTION plproxy.get_cluster_config (cluster_name text, out key text, out val text)
RETURNS SETOF record AS $$
BEGIN
    RETURN;
END;
$$ LANGUAGE plpgsql;

The details of configuration parameters and what they do can be found in the plproxy documentation.

Now the setup is complete and we can start playing around with our new cluster.
Let’s create a new table to store usernames on both partitions

#queries_0000=# CREATE TABLE users (username text PRIMARY KEY);
#queries_0001=# CREATE TABLE users (username text PRIMARY KEY);

Also we must create a new function that is used to insert new usernames into the table:

CREATE OR REPLACE FUNCTION insert_user(i_username text) RETURNS text AS $$
BEGIN
    PERFORM 1 FROM users WHERE username = i_username;
    IF NOT FOUND THEN
        INSERT INTO users (username) VALUES (i_username);
        RETURN 'user created';
    ELSE
        RETURN 'user already exists';
    END IF;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER;

Now let’s create the proxy function on the proxy database that will call the partitions

queries=#
CREATE OR REPLACE FUNCTION insert_user(i_username text) RETURNS TEXT AS $$
    CLUSTER 'queries'; RUN ON hashtext(i_username);
$$ LANGUAGE plproxy;

Filling the partitions with random data:

SELECT insert_user('user_number_'||generate_series::text) FROM generate_series(1,10000);

Now if we go to the partition databases we will see that both of them are filled
and the distribution is quite even.

queries_0001 count(*) -> 5071
queries_0000 count(*) -> 4930

To be continued…

For example if you have a column username that is used to keep the usernames as text then an average row width could be something like 24B eg. ‘mister_nice@hotmail.com’. If you have 1 million users this means you will be using 24MB just for keeping the values, and when the index is accessed regularly the database does it’s best to keep it all in the memory.

The solution itself is simple. You should create a functional index on the column that is calculated using a hash algorithm When using the PostgreSQL built-in hash function called hashtext(text) we are able to decrease the space needed to store the value from 24B to 4B (Actual index tuples have a lot more information that is used internally)

CREATE INDEX my_ix ON users (hashtext(username));

This statement will generate an index that is much more compact as it only has to store 4B per user. Raw data itself is not anymore kept inside the index when you create the functional index, instead the result given back from the hash function is stored there. For ‘mister_nice@hotmail.com’ it would be hashtext(‘mister_nice@hotmail.com’) = 1408893908 that is a 4B integer. The select statement that is used to look up the user is as follows:

SELECT * FROM users WHERE hashtext(username) = hashtext($1) AND username = $1;

Hashing can create hash collisions (different usernames will have the same hash) so that is why we need to also add the exact match criteria (username = $1). This select will use the created hashtext index to find 1..N rows that match the calculated hash and then filter out the ones whos username is an exact match.

Just a few numbers from a quick test:
100K rows 24B text each
Size of: btree(text) = 516096
Size of: btree(hashtext(text)) = 245760
This is a 52% gain.

select distinct on (key_user) * from order_status_log where key_status=7 order by key_user, deliver_time desc;
explain:
Unique  (cost=48603.10..50390.54 rows=109180 width=78)
    ->  Sort  (cost=48603.10..49496.82 rows=357487 width=78)
    Sort Key: key_user, deliver_time
        ->  Seq Scan on user_number  (cost=0.00..15629.34 rows=357487 width=78)
        Filter: (key_status = 7)

Note that the columns that are used for distinct must also be in the order by clause!

INSERT INTO tablename (column, column...) VALUES (row1_val1, row1_val2...), (row2_val1, row2_val2)..;

I did some quick tests to compare the performance difference between multiple insert statments inside one transaction versus one multirow insert. The testset was 100K records and i run the tests for several times, the magnitude of difference in performance was constant over the tests, showing that multi-row insert is approximately 4 times faster than normal insert statements. Multi-row insert needs the statment to be parsed, prepared and changes written to WAL only once therefore resulting in less overhead.

Test:

tmpdb=# create table things (things_id serial primary key, thing text);
NOTICE:  CREATE TABLE will create implicit sequence "things_things_id_seq" for serial column "things.things_id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "things_pkey" for table "things"
CREATE TABLE

Multi-row insert SQL file

insert into things (thing) values ('thing nr. 0'),
('thing nr. 1'),
('thing nr. 2'),
('thing nr. 3'),
...
('thing nr. 99999),
('thing nr. 100000);

Multiple insert statments SQL file

begin;
insert into things (thing) values ('thing nr. 0');
insert into things (thing) values ('thing nr. 1');
insert into things (thing) values ('thing nr. 2');
....
insert into things (thing) values ('thing nr. 99999');
insert into things (thing) values ('thing nr. 100000');
commit;

FALSE:
explain SELECT * FROM orders WHERE creation_date > NOW() - interval '30 mins' AND key_status = 5;
-----------------------------------------------------------------------------------
 Seq Scan on orders  (cost=0.00..9370300.50 rows=1964090 width=217)
   Filter: ((creation_date > (now() - '00:30:00'::interval)) AND (key_status = 5))

In the first case the planner sees the timestamp parameter NOW() as a inline function and therefore can’t estimate the number of rows matching the criteria as the result of a function call can never be determined (except for immutable functions). This results in an suboptimal execution plan (sequential scan)

CORRECT:
explain SELECT * FROM orders WHERE creation_date > 'NOW'::timestamptz - interval '30 mins' AND key_status = 5;
--------------------------------------------------------------------------------------------
 Index Scan using idx_order_creation_date on orders  (cost=0.00..98.03 rows=587 width=217)
   Index Cond: (creation_date > '2006-03-03 15:07:22.492173+02'::timestamp with time zone)
   Filter: (key_status = 5)

In the second case the planner handles ‘NOW’ as a constant and therefore is able to estimate how many rows have a greater value resulting in a optimal execution plan (index range scan)

Instead ‘NOW’ you can also use the variable CURRENT_TIMESTAMP and other similar constants which you don’t have to quote. They are somewhere amidst other useful information in the PostgreSQL documentation.