Tuesday, August 16, 2016

Importing PostgreSQL 9.6's SQL parser

In almost every releases of Pgpool-II, we import the latest version of PostgreSQL's SQL parser (more precisely, the raw parser). This time, our new Pgpool-II developer faced with an interesting problem while importing PostgreSQL 9.6's parser.

In PostgreSQL, the SQL parser is written in bion, a general-purpose parser generator. The particular file including the SQL grammar rules is named "gram.y". gram.y used to include "scan.l", which is a lexical scanner written in flex.

 In reality, gram.y is translated into a C source file by bison, then compiled. Same thing can be said to scan.l, which is translated by flex though.

So the main part of SQL parser  source file was single big file consisted of gram.y and scan.l.

From PostgreSQL 9.6, however, PostgreSQL developers decided to keep gram.y and flex.l separated.

Build backend/parser/scan.l and interfaces/ecpg/preproc/pgc.l standalone.

This gives enough confusion to the developer in charge of the work and took some time before realize the change. I would say it's a fun part of the task when we work on an OSS project:-)  However I cannot stop saying that it would be nice if the SQL parser is exported as a separate library so that we do not need this kind of work in every releases of Pgpool-II.

5 comments:

  1. Hi Tatsuo.

    I completely agree with you having the SQL parser as a separate library would be really helpful. Many other middleware database products may benefit from that. Why don't you propose this on -hackers?

    On a separate note: it would be awesome if you would update the post -or write a new one- detailing the steps it finally took your colleague to import the code :)

    Thanks!

    ReplyDelete
    Replies
    1. Yeah, I have once posted a proposal to export a separate library to hackers list. Tom Lane gave me "-1" because the internal of parser is not stable enough to be exported as API.

      > On a separate note: it would be awesome if you would update the post -or write a new one- detailing the steps it finally took your colleague to import the code :)

      Sure I will.

      Delete
  2. This comment has been removed by a blog administrator.

    ReplyDelete
  3. This comment has been removed by a blog administrator.

    ReplyDelete
  4. This comment has been removed by a blog administrator.

    ReplyDelete

Dynamic spare process management in Pgpool-II

Pre-fork architecture in Pgpool-II Pgpool-II uses fixed number of pre-forked child process which is responsible for accepting and handling e...