Make homepage|Add to favorites
 

MYSQL TOOLS
reviews, articles, tips & snippets

 
Buy SQL » MySQL » Dynamic pivot tables (transform rows to columns)

Dynamic pivot tables (transform rows to columns)

Author: rootsql Date: 1-11-2011, 21:50

MySQL pivot table with dynamic headers.

 

Let’s assume we have a table of properties (a properties table) – ‘properties’ (script of its creation is given below), and we need to do data transformation for the report.

+----+---------+---------------+--------+
| id | item_id | property_name | value  |
+----+---------+---------------+--------+
|  1 |       1 | color         | blue   |
|  2 |       1 | size          | large  |
|  3 |       1 | weight        | 65     |
|  4 |       2 | color         | orange |
|  5 |       2 | weight        | 57     |
|  6 |       2 | size          | large  |
|  7 |       3 | size          | small  |
|  8 |       3 | color         | red    |
|  9 |       3 | weight        | 12     |
| 10 |       4 | color         | violet |
| 11 |       4 | size          | medium |
| 12 |       4 | weight        | 34     |
| 13 |       5 | color         | green  |
| 14 |       5 | weight        | 10     |
+----+---------+---------------+--------+
=>
+---------+--------+--------+--------+
| item_id | color  | size   | weight |
+---------+--------+--------+--------+
|       1 | blue   | large  | 65     |
|       2 | orange | large  | 57     |
|       3 | red    | small  | 12     |
|       4 | violet | medium | 34     |
|       5 | green  | NULL   | 10     |
+---------+--------+--------+--------+

As is generally known (as it is known), in MySql there is no function of automatically table transformation. Surely we can use some program (tool), which is able to connect to MySql and execute data transformation. But in this case we want to do it manually, and we have one variant – to write a query which will execute the data rotation in the table.

This query can have such type as:

SELECT
  item_id,
  MAX(IF(property_name = 'color', value, NULL)) AS color,
  MAX(IF(property_name = 'size', value, NULL)) AS size,
  ...
  ...
  ...
FROM
  properties
GROUP BY
  item_id;

As you can see – for each value `property_name` we need to do definite actions. It can be easy when the types of properties do not change. But what we should do if the values of properties in the column `property_name` often change or complement with the new ones? In this case we would change the query every time. In this case the algorithm of dynamic construction of this query can help us, this algorithm has to read all possible values of the column `property_name`, and on its basement create a query. The algorithm of query construction is:

SET @sql = NULL;
SELECT
  GROUP_CONCAT(DISTINCT
  CONCAT(
  'MAX(IF(property_name = ''',
  property_name,
  ''', value, NULL)) AS ',
  CONCAT("`", property_name, "`"))) INTO @sql
FROM properties;
SET @sql = CONCAT('SELECT item_id, ', @sql, ' FROM properties GROUP BY item_id');

As the result will be created the query:

SELECT
  item_id,
  MAX(IF(property_name = 'color', value, NULL)) AS color,
  MAX(IF(property_name = 'size', value, NULL)) AS size,
  MAX(IF(property_name = 'weight', value, NULL)) AS weight
FROM
  properties
GROUP BY
  item_id

(the query is formatted additionally for readability)

 

The query is written into variable @sql; now we can execute it with prepared statements:

PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
+---------+--------+--------+--------+
| item_id | color  | size   | weight |
+---------+--------+--------+--------+
|       1 | blue   | large  | 65     |
|       2 | orange | large  | 57     |
|       3 | red    | small  | 12     |
|       4 | violet | medium | 34     |
|       5 | green  | NULL   | 10     |
+---------+--------+--------+--------+

 

Script of creation and filling of the table:

CREATE TABLE properties (
  id INT(11) NOT NULL AUTO_INCREMENT,
  item_id INT(11) DEFAULT NULL,
  property_name VARCHAR(255) DEFAULT NULL,
  value VARCHAR(255) DEFAULT NULL,
  PRIMARY KEY (id)
);
INSERT INTO properties VALUES 
  (1, 1, 'color', 'blue'),
  (2, 1, 'size', 'large'),
  (3, 1, 'weight', 65),
  (4, 2, 'color', 'orange'),
  (5, 2, 'weight', 57),
  (6, 2, 'size', 'large'),
  (7, 3, 'size', 'small'),
  (8, 3, 'color', 'red'),
  (9, 3, 'weight', 12),
  (10, 4, 'color', 'violet'),
  (11, 4, 'size', 'medium'),
  (12, 4, 'weight', 34),
  (13, 5, 'color', 'green'),
  (14, 5, 'weight', 10);

Tags: Tips

Dear visitor, you went to the site as unregistered user.
We recommend you to Register or Sign In.

Comments:

Leave a comment
Quote
  • oskar

  • 19 January 2013 06:58
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
I have found your code of great help, but I have got stuck when I trid to use it slightly differently.

basically my table looks ike this
dt        ticker         open
1            A                1.2
1            B                 6
2            A                 1.4
2            B                 6.5

I would like it to turn out to be
dt           A            B
1            1.2         1.4
2            6            6.5


but it turns out to be
dt           A            B
1            1.2         NULL
1            NULL         1.4
2            6            NULL
2            NULL            6.5


the adjusted code I have used is

SET @sql = NULL;
SELECT
  GROUP_CONCAT(DISTINCT
    CONCAT(
      '(IF(ticker = ''',
      ticker,
      ''', open, NULL)) AS ''',
      ticker,''''
    )
  ) INTO @sql
FROM
  prices;

SET @sql = CONCAT('SELECT dt, ', @sql, ' FROM prices');
-- SET @sql = CONCAT('SELECT dt, ', @sql, ' FROM prices GROUP BY dt');

  PREPARE stmt FROM @sql;
  EXECUTE stmt;



would very much appreciate if you could help me out with this.
Quote
  • Group: Administrators
  • ICQ:
  • Registration: 27.09.2011
  • Comments: 3
  • Publications: 35
^
Hi, Oskar.

You should use aggregate function, for example - MAX()

SELECT GROUP_CONCAT(DISTINCT CONCAT('MAX(IF(ticker =...
Quote
  • dp

  • 10 April 2013 11:49
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
I'm seeing sysntax error in this code.
Quote
  • Group: Chief editors
  • ICQ:
  • Registration: 8.11.2011
  • Comments: 1
  • Publications: 7
^
dp,
hmm strange. code is correct. The query is executed without error. If you have an error, specify what?
Quote
  • Jonas

  • 17 May 2013 08:25
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
I have a table 'narvarorapportering' with fields 'amne', 'lektionstillfalle', 'narvarande', 'efternamn'.

I have replaced tablename and fields in your code with my own like this:

SET @sql = NULL;
SELECT
  GROUP_CONCAT(DISTINCT
    CONCAT(
      'MAX(IF(lektionstillfalle = ''',
      lektionstillfalle,
      ''', narvarande, NULL)) AS ',
      lektionstillfalle
    )
  ) INTO @sql
FROM
  narvarorapportering;
SET @sql = CONCAT('SELECT efternamn, ', @sql, ' FROM narvarorapportering GROUP BY efternamn');

PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;



When trying this I get the following error message:

#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '36,MAX(IF(lektionstillfalle = 'v 37', narvarande, NULL)) AS v 37,MAX(IF(lektions' at line 1

Any help on what could be wrong would be really great!

/Jonas
Quote
  • Group: Administrators
  • ICQ:
  • Registration: 27.09.2011
  • Comments: 3
  • Publications: 35
^
Check that table is not empty, and lektionstillfalle column has no NULL values.
Quote
  • Jonas

  • 17 May 2013 09:30
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
...the table is not empty and the column 'lektionstillfalle' has no NULL values...

/jonas
Quote
  • admin

  • 17 May 2013 10:36
  • Group: Administrators
  • ICQ:
  • Registration: 23.01.2011
  • Comments: 5
  • Publications: 2
^
Jonas,
If the names of the future columns contains a space, then there is a syntax error.
try:

SET @sql = NULL;
SELECT
GROUP_CONCAT(DISTINCT
CONCAT(
'MAX(IF(lektionstillfalle = ''',
lektionstillfalle,
''', narvarande, NULL)) AS ',
CONCAT("`",lektionstillfalle,"`")
)
) INTO @sql
FROM
narvarorapportering;
SET @sql = CONCAT('SELECT efternamn, ', @sql, ' FROM narvarorapportering GROUP BY efternamn');

PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
Quote
  • Jonas

  • 17 May 2013 10:45
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
I really appreciate you helping me with this and for your quick reply!

When trying this I get a different error message:

#1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'FROM narvarorapportering GROUP BY efternamn' at line 1

/Jonas
Quote
  • admin

  • 17 May 2013 10:52
  • Group: Administrators
  • ICQ:
  • Registration: 23.01.2011
  • Comments: 5
  • Publications: 2
^
Jonas,
show DDL of the narvarorapportering table.
 SHOW CREATE TABLE narvarorapportering; 
Quote
  • Jonas

  • 17 May 2013 11:06
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
CREATE TABLE `narvarorapportering` (
 `IDnarvrap` int(5) NOT NULL auto_increment,
 `larare` varchar(30) character set utf8 collate utf8_swedish_ci default NULL,
 `amne` varchar(30) character set utf8 collate utf8_swedish_ci default NULL,
 `lektionstillfalle` varchar(30) character set utf8 collate utf8_swedish_ci default NULL,
 `narvarande` enum('ja','nej') character set utf8 collate utf8_swedish_ci default NULL,
 `fornamn` varchar(20) character set utf8 collate utf8_swedish_ci default NULL,
 `efternamn` varchar(20) character set utf8 collate utf8_swedish_ci default NULL,
 `langd` time NOT NULL,
 PRIMARY KEY  (`IDnarvrap`)
) ENGINE=MyISAM AUTO_INCREMENT=8134 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Quote
  • admin

  • 17 May 2013 11:13
  • Group: Administrators
  • ICQ:
  • Registration: 23.01.2011
  • Comments: 5
  • Publications: 2
^
also show result of the query:
SET @sql = NULL;
SELECT
GROUP_CONCAT(DISTINCT
CONCAT(
'MAX(IF(lektionstillfalle = ''',
lektionstillfalle,
''', narvarande, NULL)) AS ',
CONCAT("`",lektionstillfalle,"`")
)
) INTO @sql
FROM
narvarorapportering;
SET @sql = CONCAT('SELECT efternamn, ', @sql, ' FROM narvarorapportering GROUP BY efternamn');


SELECT @sql;
Quote
  • Jonas

  • 17 May 2013 11:18
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
SELECT efternamn, MAX(IF(lektionstillfalle = '36b', narvarande, NULL)) AS `36b`,MAX(IF(lektionstillfalle = '45b', narvarande, NULL)) AS `45b`,MAX(IF(lektionstillfalle = 'Vecka 36', narvarande, NULL)) AS `Vecka 36`,MAX(IF(lektionstillfalle = 'v 37', narvarande, NULL)) AS `v 37`,MAX(IF(lektionstillfalle = 'v 37:2', narvarande, NULL)) AS `v 37:2`,MAX(IF(lektionstillfalle = 'v 38', narvarande, NULL)) AS `v 38`,MAX(IF(lektionstillfalle = 'v 38:2', narvarande, NULL)) AS `v 38:2`,MAX(IF(lektionstillfalle = '35', narvarande, NULL)) AS `35`,MAX(IF(lektionstillfalle = 'v 40', narvarande, NULL)) AS `v 40`,MAX(IF(lektionstillfalle = 'v 40:2', narvarande, NULL)) AS `v 40:2`,MAX(IF(lektionstillfalle = 'v 41', narvarande, NULL)) AS `v 41`,MAX(IF(lektionstillfalle = 'v 41:2', narvarande, NULL)) AS `v 41:2`,MAX(IF(lektionstillfalle = 'v 2', narvarande, NULL)) AS `v 2`,MAX(IF(lektionstillfalle = 'v 42', narvarande, NULL)) AS `v 42`,MAX(IF(lektionstillfalle = 'v 42:2', narvarande, NULL)) AS `v 42:2`,MAX(IF(lektionstillfalle = 'v 45', narvarande, FROM narvarorapportering GROUP BY efternamn
Quote
  • admin

  • 17 May 2013 12:23
  • Group: Administrators
  • ICQ:
  • Registration: 23.01.2011
  • Comments: 5
  • Publications: 2
^
Increase the variable group_concat_max_len
for example:
SET group_concat_max_len=2048;
Quote
  • Jonas

  • 17 May 2013 12:46
  • Group: Guests
  • ICQ:
  • Registration: --
  • Comments: 0
  • Publications: 0
^
When I try as below I get no error message. The query is executed correctly.
However, it comes back empty. The query gives back 0 rows.
Again, thanks for your help!

SET group_concat_max_len=5000;
SET @sql = NULL;
SELECT
GROUP_CONCAT(DISTINCT
CONCAT(
'MAX(IF(lektionstillfalle = ''',
lektionstillfalle,
''', narvarande, NULL)) AS ',
CONCAT("`",lektionstillfalle,"`")
)
) INTO @sql
FROM
narvarorapportering;
SET @sql = CONCAT('SELECT efternamn, ', @sql, ' FROM narvarorapportering GROUP BY efternamn');

PREPARE stmt FROM @sql;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;