📜 ⬆️ ⬇️

Load data into postgresql

I read an article about overclocking the performance of insert operations in an Oracle DBMS using Java transactions. Since I am currently working with postgresql, I decided to check how much this DBMS will issue in conjunction with the golan.
Used stack: golang-1.1.2 + postgresql-9.3.1.
To run the program, you need a Golang driver for postgresql. I use the github.com/lib/pq driver, which I installed using the go get -u github.com/lib/pq command.

Testing environment


Home desktop with archlinux-x86_64, kernel-3.11.5, 4GB of RAM (at the time of the test was approximately 1.2GB free), the usual old spindle HDD. Postgres is more or less configured (shared_buffers increased to 768MB and max_connections to 1000, + some other tuning), but without hardcore.
For the purity of the experiment did not make any Goroutin - all in 1 stream.

Text of the program
/* sql_insertion_test01.go * Synthetic test for simple insert operations into postgresql database * Call: ./sql_insertion_test01 * Pls, do not use name "insert_test", because this name is always used by go-pkg-system */ package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "time" ) const DB_CONNECT_STRING = "host=localhost port=5432 user=your_role password=your_password dbname=your_database sslmode=disable" func main() { db, err := sql.Open("postgres", DB_CONNECT_STRING) defer db.Close() if err != nil { fmt.Printf("Database opening error -->%v\n", err) panic("Database error") } init_database(&db) make_insertion(&db) } /*-----------------------------------------------------------------------------*/ func init_database(pdb **sql.DB) { db := *pdb init_db_strings := []string{ "DROP SCHEMA IF EXISTS sb CASCADE;", "CREATE SCHEMA sb;", //be careful - next multiline string is quoted by backquote symbol `CREATE TABLE sb.test_data( id serial, device_id integer not null, parameter_id integer not null, value varchar(100), event_ctime timestamp default current_timestamp, constraint id_pk primary key (id));`} for _, qstr := range init_db_strings { _, err := db.Exec(qstr) if err != nil { fmt.Printf("Database init error -->%v\n", err) panic("Query error") } } fmt.Println("Database rebuilded successfully") } /*-----------------------------------------------------------------------------*/ func make_insertion(pdb **sql.DB) { db := *pdb const TEST_NUMBER = 400000 // backquotes for next multiline string const INSERT_QUERY = `insert into sb.test_data(device_id, parameter_id, value) values ($1, $2, $3);` insert_query, err := db.Prepare(INSERT_QUERY) defer insert_query.Close() if err != nil { fmt.Printf("Query preparation error -->%v\n", err) panic("Test query error") } t1 := time.Now() for i := 0; i < TEST_NUMBER; i++ { _, err = insert_query.Exec(i, i, "0") if err != nil { fmt.Printf("Query execution error -->%v\n", err) panic("Error") } } t2 := time.Since(t1) fmt.Printf("%v queries are executed for %v seconds (%v per second)\n", TEST_NUMBER, t2.Seconds(), TEST_NUMBER/t2.Seconds()) // do not forget clean up after work ) //_, err = db.Query("TRUNCATE sb.test_data;") } 



As can be seen from the text - no special allotments in the transaction, Gorotin, temporary tables and other tricks - direct input.
Well, of course, a bunch of prepare-exec.

results


Results - 16000-17000 inserts per second on my machine.
The results do not depend on the constant TEST_NUMBER (set 1000, 2000, 4000, 40,000 and stopped at 400,000). It is possible that plugging will occur at large values, but I think that there is enough of such performance for current purposes.
I expected to get the worst numbers and planned to tinker with transactions, but it was not necessary.
')

findings


I think that the test is still somewhat synthetic - just the data fit into the cache. Do not leave the feeling that with some parameters had to rest against the performance of the disk subsystem. And the desktop core allocates memory more vigorously than the server one.
It is possible that the authors of the driver use some kind of trick. But nevertheless, I think that this bundle of applications is quite usable.

Source: https://habr.com/ru/post/197718/


All Articles