Sending Data

Overview

Start at the deployment level: one QuestDB handle covers row ingestion, whole-dataframe ingestion, and queries. The handle owns the connection pools; db.sender() lends a row-building sender and db.dataframe() keeps whole sources on the direct columnar path.

import questdb
from questdb import TimestampNanos
import pandas as pd

conf = 'ws::addr=localhost:9000;'
with questdb.connect(conf) as db:
    with db.sender() as sender:
        sender.row(
            'trades',
            symbols={'symbol': 'ETH-USD', 'side': 'sell'},
            columns={'price': 2615.54, 'amount': 0.00044},
            at=TimestampNanos.now())
        sender.flush(wait=True)

    # Whole dataframes at once
    df = pd.DataFrame({
        'symbol': pd.Categorical(['ETH-USD', 'BTC-USD']),
        'side': pd.Categorical(['sell', 'sell']),
        'price': [2615.54, 39269.98],
        'amount': [0.00044, 0.001],
        'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})

    db.dataframe(df, table_name='trades', at='timestamp')

The pooled sender holds an internal QWP buffer. A successful with block publishes pending rows when it returns the lease to the pool. Row appends also auto-flush by default; the thresholds are configured on the parent handle.

Two API layers

The package has two API layers:

• Deployment level — questdb.connect() returns a QuestDB handle that addresses a whole deployment: addr lists every cluster node, the handle owns the ingestion and reader connection pools, and its methods are database operations (sender(), dataframe(), query(), reader(), server_info()).

• Connection level — the standalone Sender drives exactly one connection (one sender = one connection). It covers the legacy ILP transports (HTTP/TCP) — including ILP/HTTP transactions, which are HTTP-only — fire-and-forget QWP/UDP datagrams, and manual control of a single ws connection (progress driving, explicit buffers and draining).

Default to the deployment level; drop down only when you need a connection-level capability:

You need	Use
Pooled row ingestion, DataFrame bulk loads, queries — the default for new code	questdb.connect() → QuestDB
HTTP transactions, UDP datagrams, ILP/TCP, or manual progress / buffer control over one ws connection	standalone Sender

Over ws:: / wss:: specifically, prefer questdb.connect() and db.sender() — the pooled lease carries the full ws delivery surface (FSN receipts, ack barriers, rejection polling); the standalone ws sender exists only for manual progress mode and explicit buffer control (see QWP/WebSocket), and its dataframe() opens a fresh direct connection per call. Moving from 4.x? See the 5.0 migration guide.

You can read more on Preparing Data and Flushing.

Constructing the Sender

From Configuration

The Sender class is generally initialized from a configuration string.

from questdb import Sender

conf = 'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    ...

See the Configuration guide for more details.

From Env Variable

You can also initialize the sender from an environment variable:

export QDB_CLIENT_CONF='http::addr=localhost:9000;'

The content of the environment variable is the same configuration string as taken by the Sender.from_conf method, but moving it to an environment variable is more secure and allows you to avoid hardcoding sensitive information such as passwords and tokens in your code.

from questdb import Sender

with Sender.from_env() as sender:
    ...

Programmatic Construction

If you prefer, you can also construct the sender programmatically. See Programmatic Construction.

Preparing Data

Appending Rows

You can append as many rows as you like through QuestDB.sender. The row arguments match Sender.row.

Appending Pandas Dataframes

Use QuestDB.dataframe to ingest a Pandas dataframe directly.

This is orders of magnitude faster than appending rows one by one.

import questdb
from questdb import QuestDBError

import sys
import pandas as pd


def example(host: str = 'localhost', port: int = 9000):
    df = pd.DataFrame({
            'symbol': pd.Categorical(['ETH-USD', 'BTC-USD']),
            'side': pd.Categorical(['sell', 'sell']),
            'price': [2615.54, 39269.98],
            'amount': [0.00044, 0.001],
            'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})
    try:
        with questdb.connect(f"ws::addr={host}:{port};") as db:
            # Ingress: publish a Pandas DataFrame into QuestDB.
            db.dataframe(
                df,
                table_name='trades',  # Table name to insert into.
                symbols=['symbol', 'side'],  # Columns to be inserted as SYMBOL types.
                at='timestamp')  # Column containing the designated timestamps.

            # Egress: query QuestDB and materialise the result as Pandas.
            with db.query(
                    "SELECT x AS trade_id, "
                    "x * 10.0 AS price "
                    "FROM long_sequence(3)") as result:
                queried = result.to_pandas()
            print(queried)

    except QuestDBError as e:
        sys.stderr.write(f'Got error: {e}\n')


if __name__ == '__main__':
    example()

For how the 4.x row-buffer dataframe methods map onto the 5.0 API, see the 5.0 migration guide.

QuestDB.dataframe ingests through the pooled columnar QWP path and always uses the direct (non-store-and-forward) column sender, independent of sf_dir.

On success, the call returns only after every row has been committed. Most loads queue their batches and commit once at the end. A very large Arrow load checkpoints about every 100 batches to keep memory bounded. The client may checkpoint earlier if the connection cannot queue another batch or if a batch must be split to fit. If a later batch fails, the exception means that the load did not finish, not necessarily that no rows landed. Any already committed prefix from that call remains in the table, and retrying the entire DataFrame can duplicate it unless the table uses suitable DEDUP UPSERT KEYS.

On a transient connection failure, the client re-sends from the caller’s DataFrame only when it knows that no rows landed. Otherwise it reports the error instead of risking a blind retry.

String vs Symbol Columns

QuestDB has a concept of symbols which are a more efficient way of storing categorical data (identifiers). Internally, symbols are deduplicated and stored as integers.

When sending data, you can specify a column as a symbol by using the symbols parameter of the row or dataframe methods. String values passed through the columns parameter are stored as VARCHAR instead — use that for one-off strings that should not be interned.

Here is an example of sending a row with two symbols and two regular columns:

from questdb import Sender
import datetime

conf = 'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    sender.row(
        'trades',
        symbols={
            'symbol': 'ETH-USD', 'side': 'sell'},
        columns={
            'price': 2615.54,
            'amount': 0.00044},
        at=datetime.datetime(
            2021, 1, 1, 12, 0, 0, tzinfo=datetime.timezone.utc))

Decimal Columns

Starting with QuestDB server version 9.2.0, you can ingest data into the database’s native DECIMAL(precision, scale) column type. This is useful when you need exact precision for financial calculations or other scenarios where floating-point rounding errors are unacceptable.

Decimal ingestion requires protocol version 3 (must be configured explicitly for TCP/TCPS). Unlike other column types, DECIMAL columns cannot be auto-created and must be pre-created with the appropriate DECIMAL(precision, scale) definition. See the QuestDB DECIMAL documentation and troubleshooting guide for more details.

To send decimal values, use Python’s decimal.Decimal type in the row method or DataFrames (decimal support is negotiated automatically over ws / wss):

from decimal import Decimal

import pandas as pd
import questdb
from questdb import TimestampNanos

# CREATE TABLE prices (
#     symbol SYMBOL,
#     price DECIMAL(18, 6),
#     timestamp TIMESTAMP_NS
# ) TIMESTAMP(timestamp) PARTITION BY DAY;

with questdb.connect('ws::addr=localhost:9000;') as db:
    with db.sender() as sender:
        sender.row(
            'prices',
            symbols={'symbol': 'BTC-USD'},
            columns={'price': Decimal('50123.456789')},
            at=TimestampNanos.now())
        sender.flush(wait=True)

    db.dataframe(
        pd.DataFrame({
            # Categorical (or string[pyarrow]) columns can be sent as
            # SYMBOL on the columnar path.
            'symbol': pd.Categorical(['BTC-USD', 'ETH-USD']),
            'price': [Decimal('50123.456789'), Decimal('2615.123456')],
        }),
        table_name='prices',
        symbols=['symbol'],
        at=TimestampNanos.now())

Loading object-dtype Decimal columns through db.dataframe(...) requires pyarrow: each such column is re-exported through Arrow to infer its decimal width and scale.

When using pandas DataFrames, you can also use PyArrow decimal types for better performance:

from decimal import Decimal
import pandas as pd
import pyarrow as pa

df = pd.DataFrame({
    'symbol': pd.Categorical(['BTC-USD', 'ETH-USD']),
    'price': pd.Series([Decimal('50123.456789'), Decimal('2615.123456')],
                       dtype=pd.ArrowDtype(pa.decimal128(12, 6)))
})

UUID, IPV4, BINARY, CHAR, DATE, LONG256 and GEOHASH Columns

Starting with QuestDB server version 10.0.0 – the first production QWP implementation – a QWP sender (udp::, ws:: or wss::) can write these seven column types. They are auto-created like any other column, except that a GEOHASH column’s precision is fixed when the column is created, so a table holding one is usually created up front.

Row by row

UUID, IPV4 and BINARY are written with the Python types that already mean them — uuid.UUID, ipaddress.IPv4Address, and bytes / bytearray / memoryview. The other four need a wrapper, because the Python type they would otherwise arrive as already means a different QuestDB column type: a str is VARCHAR, an int is LONG, and a datetime is TIMESTAMP. The wrappers are Char, DateMillis, Long256 and Geohash.

import datetime
import ipaddress
import sys
import time
import uuid

import questdb
from questdb import (
    Char,
    DateMillis,
    Geohash,
    Long256,
    QuestDBError,
    TimestampNanos,
)


def _query_when_applied(db, sql, attempts=100):
    """Rows land through the WAL, so a read issued straight after the
    write acknowledges can still be ahead of them."""
    frame = db.query(sql).to_pandas()
    for _ in range(attempts):
        if not frame.empty:
            break
        time.sleep(0.05)
        frame = db.query(sql).to_pandas()
    return frame


def example(
        host: str = 'localhost',
        port: int = 9000,
        table_name: str = 'devices'):
    with questdb.connect(f'ws::addr={host}:{port};') as db:
        # These column types are auto-created like any other, but a
        # GEOHASH column's precision is fixed when the column is
        # created, so this table states it up front.
        db.execute(
            f'CREATE TABLE IF NOT EXISTS {table_name} ('
            '  device_id UUID,'
            '  address IPV4,'
            '  payload BINARY,'
            '  grade CHAR,'
            '  last_seen DATE,'
            '  checksum LONG256,'
            '  location GEOHASH(5c),'
            '  timestamp TIMESTAMP_NS'
            ') TIMESTAMP(timestamp) PARTITION BY DAY WAL')

        with db.sender() as sender:
            sender.row(
                table_name,
                columns={
                    # UUID, IPV4 and BINARY are written with the
                    # ordinary Python types.
                    'device_id': uuid.UUID(
                        '123e4567-e89b-12d3-a456-426614174000'),
                    'address': ipaddress.IPv4Address('10.0.0.7'),
                    'payload': b'\x00\x01\x02\x03',
                    # The rest need a wrapper, because the Python
                    # type they would otherwise arrive as already
                    # means another QuestDB column type.
                    'grade': Char('A'),
                    # A DATE is a millisecond timestamp rather than
                    # a civil date; `DateMillis.now()` and
                    # `DateMillis(millis)` build one too.
                    'last_seen': DateMillis.from_datetime(
                        datetime.datetime(
                            2024, 1, 2, 3, 4, 5, 678000,
                            tzinfo=datetime.timezone.utc)),
                    'checksum': Long256(0xdeadbeef),
                    # 5 characters of base32, so 25 bits of
                    # precision -- the GEOHASH(5c) the column holds.
                    'location': Geohash.from_string('u33d8'),
                },
                at=TimestampNanos.now())
            sender.flush(wait=True)

        print(_query_when_applied(
            db,
            'SELECT device_id, address, grade, last_seen, checksum, '
            f'location FROM {table_name} LIMIT -1'))


if __name__ == '__main__':
    try:
        example()
    except QuestDBError as e:
        sys.stderr.write(
            f'Got error: {e} (code={e.code}, in_doubt={e.in_doubt})\n')
        sys.exit(1)

A few things worth knowing about the wrappers:

• DateMillis is a millisecond timestamp, not a civil date — that is what QuestDB DATE holds. Build it from milliseconds, from a datetime.datetime with DateMillis.from_datetime, or from the clock with DateMillis.now().

• Geohash carries bits and precision. Geohash.from_string('u33d8') packs one to twelve base32 characters at five bits each. Within one buffer’s worth of rows a column’s precision is fixed by the first row that reaches it, and a later row at a different precision is rejected by Sender.row itself. A flush clears the pin, so a precision the server’s column does not have comes back from the server rather than being caught locally.

• Long256 takes an unsigned integer below 2**256.

• Char takes exactly one UTF-16 code unit, so a character outside the Basic Multilingual Plane does not fit — use a str (VARCHAR) for those.

DataFrames

A DataFrame states these types through the column’s own Arrow type where the type is specific enough to say so, and through schema_overrides where it is not:

Column type	Arrow type	schema_overrides
UUID	pa.uuid()	'uuid'
LONG256	—	'long256'
IPV4	—	'ipv4' on a uint32 column
CHAR	—	'char' on a uint16 column
GEOHASH	—	('geohash', bits) on an integer column, signed or unsigned
DATE	pa.timestamp('ms'), pa.date32(), pa.date64()	—
BINARY	pa.binary(), pa.large_binary(), pa.binary(n)	'binary' on a 16- or 32-byte pa.binary(n) column

A binary column written as UUID holds each value’s uuid.UUID.bytes, in RFC 4122 (big-endian) order. One written as LONG256 holds the 256-bit value little-endian, least-significant byte first: the bytes of value.to_bytes(32, 'little').

schema_overrides needs a fully Arrow-backed frame: every column an ArrowDtype, as df.convert_dtypes(dtype_backend='pyarrow') makes it. A frame with any NumPy, object or masked column refuses the argument.

A 16- or 32-byte pa.binary(n) column needs a label to be a UUID or a LONG256: pa.uuid(), questdb.column_type field metadata of uuid or long256, schema_overrides, or a df.attrs['questdb'] claim, and schema_overrides={'col': 'binary'} or questdb.column_type metadata of binary labels it as plain bytes. Another extension type or questdb.column_type value is not a label. Without one, what happens depends on where the column comes from. From a pa.Table, a pa.RecordBatch or another native Arrow source it is written as BINARY, and the questdb logger records a warning naming the column once the write is done. In a pandas frame, Arrow-backed or not, it is refused, because a pandas ArrowDtype holds no field metadata and the label may have been lost on the way in. Variable-length binary, pa.binary(), is BINARY either way and needs no label.

import sys
import time
import uuid

import pandas as pd
import pyarrow as pa

import questdb
from questdb import QuestDBError, TimestampNanos


def _query_when_applied(db, sql, attempts=100):
    """Rows land through the WAL, so a read issued straight after the
    write acknowledges can still be ahead of them."""
    frame = db.query(sql).to_pandas()
    for _ in range(attempts):
        if not frame.empty:
            break
        time.sleep(0.05)
        frame = db.query(sql).to_pandas()
    return frame


def example(
        host: str = 'localhost',
        port: int = 9000,
        table_name: str = 'devices_df'):
    with questdb.connect(f'ws::addr={host}:{port};') as db:
        db.execute(
            f'CREATE TABLE IF NOT EXISTS {table_name} ('
            '  device_id UUID,'
            '  address IPV4,'
            '  grade CHAR,'
            '  last_seen DATE,'
            '  checksum LONG256,'
            '  location GEOHASH(5c),'
            '  timestamp TIMESTAMP_NS'
            ') TIMESTAMP(timestamp) PARTITION BY DAY WAL')

        device_ids = [
            uuid.UUID('123e4567-e89b-12d3-a456-426614174000'),
            uuid.UUID('00000000-0000-0000-0000-0000000000ff')]
        frame = pd.DataFrame({
            # A DataFrame states a type through the column's own
            # Arrow type where the type is specific enough to say
            # so, and through `schema_overrides` where it is not.
            # Only `pa.uuid()` names a type outright here: a
            # fixed-size binary of 16 or 32 bytes does not say
            # whether it holds a UUID, a LONG256 or plain bytes, and
            # a DataFrame refuses one that nothing labels; uint32 and
            # uint16 are plain integers. Each of those is named
            # below.
            'device_id': pd.Series(
                [value.bytes for value in device_ids],
                dtype=pd.ArrowDtype(pa.binary(16))),
            'address': pd.Series(
                [0x0A000007, 0x0A000008],
                dtype=pd.ArrowDtype(pa.uint32())),
            'grade': pd.Series(
                [ord('A'), ord('B')],
                dtype=pd.ArrowDtype(pa.uint16())),
            # A millisecond Arrow timestamp is itself the DATE
            # claim; `pa.date32()` and `pa.date64()` work too.
            'last_seen': pd.Series(
                [1704164645678, 1704164645679],
                dtype=pd.ArrowDtype(pa.timestamp('ms'))),
            'checksum': pd.Series(
                [(0xdeadbeef).to_bytes(32, 'little'),
                 (1).to_bytes(32, 'little')],
                dtype=pd.ArrowDtype(pa.binary(32))),
            # A GEOHASH rides on a signed integer wide enough for
            # its precision, and `schema_overrides` names the bits.
            # 27364744 is the base32 string 'u33d8' packed into 25
            # bits -- `Geohash.from_string` does that conversion for
            # the row API.
            'location': pd.Series(
                [27364744, 27364745],
                dtype=pd.ArrowDtype(pa.int32())),
        })
        db.dataframe(
            frame,
            table_name=table_name,
            schema_overrides={
                'device_id': 'uuid',
                'address': 'ipv4',
                'grade': 'char',
                'checksum': 'long256',
                'location': ('geohash', 25),
            },
            at=TimestampNanos.now())

        # Reading a result back and writing it out again keeps
        # every one of these types. A pandas dtype holds an Arrow
        # type and no field, so the claim travels in
        # `df.attrs['questdb']`, which `dataframe()` reads. It works
        # the same on all three `to_pandas` backends, and needs no
        # `schema_overrides` second time round.
        read_back = _query_when_applied(
            db, f'SELECT * FROM {table_name} LIMIT -2')
        print(read_back.attrs['questdb'])

        # Written back into the table it came from: the server
        # refuses a column whose type does not match the one the
        # table holds, so this append landing is the claim doing its
        # job. Written to a new table name instead, the same frame
        # auto-creates it with these types rather than with BINARY
        # and plain integers.
        db.dataframe(read_back, table_name=table_name, at='timestamp')

        print(_query_when_applied(
            db,
            'SELECT device_id, address, grade, last_seen, checksum, '
            f'location FROM {table_name}'))


if __name__ == '__main__':
    try:
        example()
    except QuestDBError as e:
        sys.stderr.write(
            f'Got error: {e} (code={e.code}, in_doubt={e.in_doubt})\n')
        sys.exit(1)

Warning

Bulk GEOHASH ingestion treats each integer as a raw bit pattern. It validates that the declared precision is in 1..=60 and fits the signed integer carrier, but it does not verify that an individual value fits that precision. For example, a value of 32 needs six bits; declaring it as GEOHASH(5b) is accepted.

If the declared precision is wrong for the encoded data, the write can succeed and QuestDB can store a different GEOHASH location or NULL. Only the low ceil(precision / 8) bytes are encoded, and the exact result of an inconsistent value is unspecified. This is a semantic data-integrity risk, not a memory-safety risk for otherwise valid NumPy/Arrow buffers. Validate bulk values before sending them, or use Geohash on the row path for strict per-value validation. Malformed custom Arrow C Data structures remain invalid input.

Reading a query result back and writing it out again keeps all of these types. A pandas dtype holds an Arrow type and no field, so the claim travels in df.attrs['questdb'], which QuestDB.dataframe reads — no second round of schema_overrides. All three to_pandas backends round-trip, in whatever shape they hand the columns back. A column you dropped, renamed or retyped simply loses its claim, and symbols / schema_overrides outrank it.

A pa.Table from to_arrow() carries the claim in each field’s questdb.column_type metadata instead, under a deliberately stricter rule. If you change a column’s type and keep its old field, for example with field.with_type(...) or pa.concat_tables(..., promote_options='permissive'), a claim the new type cannot carry fails the write with QuestDBErrorCode.ArrowIngest before anything is sent. Drop the field’s metadata with field.remove_metadata() once you retype it. QuestDB.dataframe explains why the two routes differ.

Populating Designated Timestamps

The at parameter of the row and dataframe methods is used to specify the designated timestamp of the rows. The designated timestamp column determines the order in which data is stored as rows and is used for partitioning.

Set by client

It can be either a TimestampNanos object, a TimestampMicros object or a datetime.datetime object.

In case of dataframes you can also specify the timestamp column name or index. If so, the column type should be a Pandas datetime64, with or without timezone information.

QuestDB stores timestamps as either microseconds (TIMESTAMP QuestDB column type) or nanoseconds (TIMESTAMP_NS QuestDB column type) as a numeric value from unix epoch in UTC. Any timezone information is dropped when sent to the database.

Note

Naive (timezone-unaware) timestamps are interpreted as UTC everywhere: DataFrame column cells, scalar at= values, row() values and query binds. Beware that datetime.now() is your local wall clock — for “now”, use TimestampNanos.now() or datetime.now(timezone.utc).

Note

Nanosecond timestamp support is only available from QuestDB 9.1.0 onwards.

Set by server

If you prefer, you can specify at=ServerTimestamp which will instruct QuestDB to set the timestamp on your behalf for each row as soon as it’s received by the server.

import questdb
from questdb import ServerTimestamp

with questdb.connect('ws::addr=localhost:9000;') as db:
    with db.sender() as sender:
        sender.row(
            'trades',
            symbols={'symbol': 'ETH-USD', 'side': 'sell'},
            columns={'price': 2615.54, 'amount': 0.00044},
            at=ServerTimestamp)  # Legacy feature, not recommended.
        sender.flush(wait=True)

Warning

Using ServerTimestamp is not recommended as it removes the ability for QuestDB to deduplicate rows and is considered a legacy feature.

Flushing

The sender accumulates data into an internal buffer. Calling Sender.flush will send the buffered data to QuestDB, and clear the buffer.

Flushing can be done explicitly or automatically.

Explicit Flushing

An explicit call to Sender.flush will send any pending data immediately.

import questdb
from questdb import TimestampNanos

with questdb.connect('ws::addr=localhost:9000;') as db:
    with db.sender() as sender:
        sender.row(
            'trades',
            symbols={'symbol': 'ETH-USD', 'side': 'sell'},
            columns={'price': 2615.54, 'amount': 0.00044},
            at=TimestampNanos.now())
        sender.flush()
        sender.row(
            'trades',
            symbols={'symbol': 'BTC-USD', 'side': 'sell'},
            columns={'price': 39269.98, 'amount': 0.001},
            at=TimestampNanos.now())
        sender.flush()

Note that the last sender.flush() is entirely optional as the pooled sender publishes any pending rows when the with block returns it to the pool (pass wait=True on the final flush to also await the server acknowledgement).

Auto-flushing

Both the standalone Sender and pooled QWP row senders auto-flush by default to avoid accumulating large or stale batches. Their defaults reflect the different protocols:

• A standalone sender publishes at 75,000 rows over HTTP, or 600 rows over TCP and QWP, and after 1 second. Over QWP/WebSocket it also publishes when its estimated encoded size reaches 90% of the current frame limit, as a pooled sender does.

• A pooled QWP sender publishes at 1,000 rows, after 100 milliseconds, or when its estimated encoded size reaches 90% of the current frame limit. Until the server advertises its limit, both senders plan for the limit a QuestDB server has by default, 2 MiB less 14 bytes, or for the local queue limit if that is lower, so rows buffered meanwhile still fit a default server once its limit is known. Explicit byte thresholds above that 90% limit are clamped. The estimate counts each symbol value at its largest encoded size, so it is not below the size of the encoded frame, unless store-and-forward has fallen back to sending full symbol dictionaries. The exact encoded size is still checked when publishing.

Over QWP/WebSocket the frame limit is the smallest of max_buf_size, the server’s advertised limit and the store-and-forward segment payload capacity. Every auto-flushed frame fits it. A buffer whose estimate reaches the full limit is published even with auto_flush_bytes=off, and when its estimate is past the limit or the whole buffer does not fit in one frame, the rows buffered before the newest one go out as a frame of their own and the newest row starts the next batch.

Auto-flushing is triggered when:

• appending a row to the internal sender buffer

• and the buffer reaches its row or byte threshold, or its interval has elapsed (there is no background timer).

Here is an example configuration string that sets up a sender to flush every 10 rows and disables the interval-based auto-flushing logic.

http::addr=localhost:9000;auto_flush_rows=10;auto_flush_interval=off;

The equivalent pooled configuration uses QWP/WebSocket. The mode belongs to the QuestDB handle and is shared by all leases; the interval starts when the first row enters an empty lease buffer.

ws::addr=localhost:9000;auto_flush_rows=10;auto_flush_interval=off;

A pooled auto-flush only publishes into the store-and-forward path; it does not wait for an acknowledgement. Keep acknowledgement barriers explicit with sender.flush(wait=True) or sender.wait(). Errors encountered while auto-publishing propagate from PooledSender.row().

On both QWP/WebSocket senders, whatever row() raises, the row was not added. BatchTooLarge means either it does not fit in a frame on its own, or the rows buffered before it do not fit even without it, which can happen when the frame limit shrinks, for example on reconnect, or when a lazily connected sender learns that the server’s limit is below the default; those rows are then discarded, and the error says how many. After any other auto-publish error a pooled sender keeps the rows buffered before it, and a standalone one clears them, as its flush() does.

If every explicit flush_and_get_fsn() must return the receipt for one complete application batch, configure auto_flush=off; otherwise an earlier auto-flush may have already published and cleared some or all of that batch. On a standalone QWP/WebSocket sender, turning off auto_flush_rows and auto_flush_interval is not enough: its byte trigger stays on.

Here is a configuration string with auto-flushing completely disabled:

http::addr=localhost:9000;auto_flush=off;

For a pooled sender, use:

ws::addr=localhost:9000;auto_flush=off;

See the Auto-flushing section for more details, and note that auto_flush_interval does NOT start a timer.

Error Reporting

TL;DR: QWP/WebSocket has the richest error reporting; among the legacy ILP transports, use HTTP.

Over QWP/WebSocket every server rejection is delivered as a structured SenderError — pushed to the pool’s error_handler (or logged through the questdb logger by default), while terminal rejections also raise QuestDBServerRejectionError from the affected sender. See QWP/WebSocket.

For the legacy ILP transports, the sender will do its best to check for errors before sending data to the server.

When using the HTTP protocol, the server will send back an error message if the data is invalid or if there is a problem with the server. This will be raised as an QuestDBError exception.

The HTTP layer will also attempt retries, configurable via the retry_timeout parameter.

When using the TCP protocol errors are not sent back from the server and must be searched for in the logs. See the Errors during flushing section for more details.

HTTP Transactions

When using the HTTP protocol, the sender can be configured to send a batch of rows as a single transaction.

Transactions are limited to a single table.

from questdb import Sender, TimestampNanos

conf = 'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    with sender.transaction('trades') as txn:
        txn.row(
            symbols={'symbol': 'ETH-USD', 'side': 'sell'},
            columns={'price': 2615.54, 'amount': 0.00044},
            at=TimestampNanos.now())
        txn.row(
            symbols={'symbol': 'BTC-USD', 'side': 'sell'},
            columns={'price': 39269.98, 'amount': 0.001},
            at=TimestampNanos.now())

The table name is set once on the transaction; txn.row() takes no table argument.

If auto-flushing is enabled, any pending data will be flushed before the transaction is started.

Auto-flushing is disabled during the scope of the transaction.

The transaction is automatically completed at the end of the with block.

• If the there are no errors, the transaction is committed and sent to the server without delays.

• If an exception is raised with the block, the transaction is rolled back and the exception is propagated.

You can also terminate a transaction explicitly by calling the commit or the rollback methods.

While transactions that span multiple tables are not supported by QuestDB, you can reuse the same sender for multiple tables.

You can also create parallel transactions by creating multiple sender objects across multiple threads.

Table and Column Auto-creation

When sending data to a table that does not exist, the server will create the table automatically.

This also applies to columns that do not exist.

The server will use the first row of data to determine the column types.

If the table already exists, the server will validate that the columns match the existing table.

If you’re using QuestDB enterprise you might need to grant further permissions to the authenticated user.

CREATE SERVICE ACCOUNT ingest;
GRANT ilp, create table TO ingest;
GRANT add column, insert ON all tables TO ingest;
--  OR
GRANT add column, insert ON table1, table2 TO ingest;

Read more setup details in the Enterprise quickstart and the role-based access control guides.

Good Practices

Create tables in advance

If you’re not happy with the default table auto creation logic, create the tables in advance. This will allow you to:

• Specify the column types explicitly.

• Configure de-duplication rules for the table.

Specify your own timestamps

Always specify your own timestamps using the at parameter.

If you use the ServerTimestamp option, QuestDB will not be able to deduplicate rows, should you ever need to send them again.

Instead, if you don’t have a timestamp immediately available, use TimestampNanos.now() to set the timestamp to the current time.

This is lighter-weight than using a fully-fledged datetime.datetime object.

Default to the deployment level

New code should connect through questdb.connect(): QWP/WebSocket combines acknowledged delivery, structured server rejections, queries, and connection pooling in one handle. Drop to the connection-level Sender when you need one of its point-to-point capabilities (see Two API layers): ILP/HTTP for transactions and the best error reporting among the legacy ILP transports, QWP/UDP for fire-and-forget, lowest-latency ingestion that can tolerate potential data loss, or ILP/TCP for servers that predate ILP/HTTP support.

Reuse the connection-owning object

Over QWP/WebSocket the long-lived object is the QuestDB handle: create one per process and share it across threads — it owns the connection pools. Sender leases are cheap by design: borrow one per unit of work (one per thread), flush, and leave the with block so the pooled connection is returned.

import questdb

db = questdb.connect('ws::addr=localhost:9000;')  # long-lived

def handle_batch(rows):
    # Short-lived lease over a pooled, already-open connection.
    with db.sender() as sender:
        for table, symbols, columns, at in rows:
            sender.row(table, symbols=symbols, columns=columns, at=at)
        sender.flush(wait=True)

At the connection level there is no pool: create longer-lived standalone Sender objects and reuse them across requests instead of constructing one per request.

from questdb import Sender

conf = 'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    # Use the sender object for multiple requests
    sender.row(...)
    sender.row(...) # remember auto-flush may trigger after any row
    sender.row(...)
    sender.flush() # you can flush explicitly at any point too
    # ...
    sender.row(...)
    sender.dataframe(...) # auto-flush may trigger within a dataframe too
    sender.flush()

Use transactions

Use transactions if you want to ensure that a group of rows is sent as a single transaction.

This feature will guarantee that the rows are sent to the server as one, even if you’re using auto-flushing.

Tune for Performance

If you need better performance:

• Tune for larger batches of rows. Tweak the auto-flush settings, or call Sender.flush less frequently.

• Use the QuestDB.dataframe method to send dataframes instead of appending rows one by one.

• Try multi-threading: The Sender logic is designed to release the Python GIL whenever possible, so you should notice an uplift in performance if you were bottlenecked by network I/O.

• Avoid sending data which is very much out of order: The server will re-order data by timestamp as it arrives. This is generally cheap for data that only affects the recent past, but if you are sending data that is very much out of order (for example, from different days), you may want to consider re-ordering it before sending. For bulk data uploads of historical data, consider using the CSV import feature for best performance.

Advanced Usage

Independent Buffers (legacy)

Buffers are managed internally by senders and are not part of the top-level 5.0 API. Legacy ILP/HTTP and ILP/TCP code that builds buffers explicitly and flushes them with sender.flush(buffer) — including the multi-database flush(buf, clear=False) fan-out pattern — keeps working through the deprecated questdb.ingress compatibility shim:

from questdb.ingress import Buffer, Sender, TimestampNanos

buf = Buffer(protocol_version=2)
buf.row(
    'trades',
    symbols={'symbol': 'ETH-USD', 'side': 'sell'},
    columns={'price': 2615.54, 'amount': 0.00044},
    at=TimestampNanos.now())

# Pin the sender to the buffer's protocol version: HTTP would otherwise
# negotiate a newer one and refuse the explicitly-versioned buffer.
conf = 'http::addr=localhost:9000;protocol_version=2;'
with Sender.from_conf(conf) as sender:
    sender.flush(buf, transactional=True)

The transactional parameter is optional and defaults to False. When set to True, the buffer is guaranteed to be committed as a single transaction, but must only contain rows for a single table.

For new code, decouple serialization from sending by borrowing one sender per thread from a questdb.connect() pool instead of sharing buffers.

Threading Considerations

A sender object is not thread-safe, but can be shared between threads if you take care of exclusive access (such as using a lock) yourself.

The QuestDB handle is the object designed to be shared: create one per process and call it from any number of threads. Everything it hands out follows the sender’s rule instead. A sender lease and a query result are each used by one thread at a time, and may be handed to another thread with proper synchronization, provided no two threads call into the same one at once. A reader lease stays on the thread that borrowed it. A sender lease holds its own lock for the whole of each call, so a second thread calling into it waits until the running call returns. Code that runs inside a lease’s call – a column value’s conversion, the Arrow stream dataframe() reads, or a handler on the questdb logger, which dataframe() logs its warnings to – therefore must not wait on another thread that uses the same lease.

With greenlets – gevent or eventlet monkey-patching, or greenlet used directly – all greenlets on one OS thread count as a single caller of the QuestDB handle and its leases. A call’s native work holds its OS thread, so another greenlet runs during a call only when code inside it yields or switches – for example, a logging handler that sends log records over the network, or an Arrow stream that reads from a socket. While such a greenlet is suspended inside a call on the handle or on one of its leases, QuestDB.close() from another greenlet on the same thread is refused with QuestDBErrorCode.InvalidApiCall, as if it had been made from inside that call, rather than waiting. Call close() once the in-flight work has finished, or call it again after the refusal.

The simplest concurrency rule: borrow (or create) one sender per thread. With QWP/WebSocket, QuestDB.sender makes this cheap — each borrow leases a pooled connection.

Close every lease, or leave it through a with block. A sender lease that is garbage-collected without close() returns its connection to the pool without sending the rows still buffered in it, and logs how many it discarded through the questdb logger at WARNING.

Notice that the questdb python module is mostly implemented in native code and is designed to release the Python GIL whenever possible, so you can expect good performance in multi-threaded scenarios.

As an example, appending a dataframe to a buffer releases the GIL (unless any of the columns reference python objects).

All network activity also fully releases the GIL.

Optimising HTTP Performance

The sender’s network communication is implemented in native code and thus does not require access to the GIL, allowing for true parallelism when used using multiple threads.

For simplicity of design and best error feedback, the .flush() method blocks until the server has acknowledged the data.

If you need to send a large number of smaller requests (in other words, if you need to flush very frequently) or are in a high-latency network, you can significantly improve performance by creating and sending using multiple sender objects in parallel.

from questdb import Sender, TimestampNanos
import pandas as pd
from concurrent.futures import ThreadPoolExecutor
import datetime

def send_data(df):
    conf_string = 'http::addr=localhost:9000;'
    with Sender.from_conf(conf_string) as sender:
        sender.dataframe(
            df,
            table_name='trades',
            symbols=['symbol', 'side'],
            at='timestamp')

dfs = [
        pd.DataFrame({
        'symbol': pd.Categorical(['ETH-USD', 'BTC-USD']),
        'side': pd.Categorical(['sell', 'sell']),
        'price': [2615.54, 39269.98],
        'amount': [0.00044, 0.001],
        'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])}
        ),
        pd.DataFrame({
        'symbol': pd.Categorical(['BTC-USD', 'BTC-USD']),
        'side': pd.Categorical(['buy', 'sell']),
        'price': [39268.76, 39270.02],
        'amount': [0.003, 0.010],
        'timestamp': pd.to_datetime(['2021-01-03', '2021-01-03'])}
        ),
]

with ThreadPoolExecutor() as executor:
    futures = [executor.submit(send_data, df)
        for df in dfs]
    for future in futures:
        future.result()

For maximum performance you should also cache the sender objects and reuse them across multiple requests, since internally they maintain a connection pool.

Sender Lifetime Control

Instead of using a with Sender .. as sender: block you can also manually control the lifetime of the sender object.

from questdb import Sender

conf = 'http::addr=localhost:9000;'
sender = Sender.from_conf(conf)
sender.establish()
# ...
sender.close()

The establish method needs to be called exactly once, but the close method is idempotent and can be called multiple times.

Table and Column Names

The client will validate table and column names while constructing the buffer.

Table names and column names must not be empty and must adhere to the following:

Table Names

Cannot contain the following characters: ?, ,, ', ", \, /, :, ), (, +, *, %, ~, carriage return (\r), newline (\n), null character (\0), and Unicode characters from \u{0001} to \u{000F} and \u{007F}. Additionally, the Unicode character for zero-width no-break space (UTF-8 BOM, \u{FEFF}) is not allowed.

A dot (.) is allowed except at the start or end of the name, and cannot be consecutive (e.g., valid.name is valid, but .invalid, invalid., and in..valid are not).

Column Names

Cannot contain the following characters: ?, ., ,, ', ", \, /, :, ), (, +, -, *, %, ~, carriage return (\r), newline (\n), null character (\0), and Unicode characters from \u{0001} to \u{000F} and \u{007F}. Like table names, the Unicode character for zero-width no-break space (UTF-8 BOM, \u{FEFF}) is not allowed.

Unlike table names, a dot (.) is not allowed in column names at all.

Programmatic Construction

Sender Constructor

You can also specify the configuration parameters programmatically:

from questdb import Sender, Protocol
from datetime import timedelta

with Sender(Protocol.Tcp, 'localhost', 9009,
        auto_flush=True,
        auto_flush_interval=timedelta(seconds=10)) as sender:
    ...

See the Configuration section for a full list of configuration parameters: each configuration parameter can be passed as named arguments to the constructor.

Python type mappings:

• Parameters that require strings take a str.

• Parameters that require numbers can also take an int.

• Millisecond durations can take an int or a datetime.timedelta.

• Any 'on' / 'off' / 'unsafe_off' parameters can also be specified as a bool.

• Paths can also be specified as a pathlib.Path.

Note

The constructor arguments have changed between 1.x and 2.x. If you are upgrading, take a look at the changelog.

Customising .from_conf() and .from_env()

If you want to further customise the behaviour of the .from_conf() or .from_env() methods, you can pass additional parameters to these methods. The parameters are the same as the ones for the Sender constructor, as documented above.

For example, here is a configuration string that is loaded from an environment variable and then customised to specify a 10 second auto-flush interval:

export QDB_CLIENT_CONF='http::addr=localhost:9000;'

from questdb import Sender, Protocol
from datetime import timedelta

with Sender.from_env(auto_flush_interval=timedelta(seconds=10)) as sender:
    ...

Protocol Version

Explicitly specifies the version of InfluxDB Line Protocol to use for sender.

Valid options are:

• protocol_version=1

• protocol_version=2

• protocol_version=3

• protocol_version=auto (default, if unspecified)

Behavior details:

Value	Behavior
1	Plain text serialization Compatible with InfluxDB servers No array type support
2	Binary encoding for f64 Full support for array requires QuestDB server version 9.0.0 or higher
3	Decimal support requires QuestDB server version 9.2.0 or higher
auto	HTTP/HTTPS: Auto-detects server capability during handshake (supports version negotiation) TCP/TCPS: Defaults to version 1 for compatibility

Here is a configuration string with protocol_version=3 for TCP:

tcp::addr=localhost:9000;protocol_version=3;

See the Protocol Version section for more details.

Which protocol?

The sender supports tcp, tcps, http, https, udp, ws, and wss protocols.

Default to the deployment level: connect through questdb.connect() over QWP/WebSocket (ws / wss) for acknowledged delivery, structured server rejections, queries and connection pooling in one handle. The other protocols belong to the connection-level standalone Sender (see Two API layers); among the legacy ILP transports, prefer ILP/HTTP for better error feedback and transaction control.

QWP/WebSocket

QWP/WebSocket (ws, or wss for TLS) is an acknowledged streaming transport. For general ingestion use questdb.connect() and db.sender() — the pooled path described throughout this guide, whose lease carries the same delivery surface: the FSN receipts (flush_and_get_fsn, flush_and_keep_and_get_fsn, published_fsn / acked_fsn, await_acked_fsn) and rejection polling (poll_error, error_events_dropped). A standalone Sender.from_conf('ws::...') additionally offers manual progress mode and explicit buffer control, described below.

Each flush publishes a frame identified by a monotonically increasing frame sequence number (FSN); the server acknowledges frames as it durably applies them, so the client can confirm delivery.

• Confirming delivery. Sender.flush_and_get_fsn flushes and returns the FSN of the published frame; Sender.flush_and_keep_and_get_fsn does the same without clearing the buffer. Sender.await_acked_fsn blocks until a given FSN is acknowledged (or a timeout elapses), and Sender.acked_fsn / Sender.published_fsn report progress without blocking.

• Progress modes. With the default qwp_ws_progress=background, acknowledgements are progressed on a background thread. With qwp_ws_progress=manual, the application must call Sender.drive_once (or one of the flush/await methods) to pump the connection.

• Server diagnostics. Per-frame server feedback is delivered to the error_handler callback, or polled via Sender.poll_error as SenderError values (Sender.error_events_dropped reports how many were dropped when no handler kept up). A diagnostic with a terminal policy halts the sender: the next sender call raises QuestDBServerRejectionError. The handler must not call back into the same sender, must be cheap and non-blocking, and — under qwp_ws_progress=background — may run on a background thread. Most such calls, from this handler or the sender’s connection_listener, raise InvalidApiCall, row() among them while auto-flush is on. Without an error_handler the sender logs rejections to the questdb logger from the same callback, so a logging handler must not write through this sender either. The pooled QuestDB handle delivers the same diagnostics through the error_handler passed to questdb.connect(), on a dedicated dispatcher thread.

• Full symbol dictionary. QuestDBErrorCode.SymbolDictFull means that a ws/wss connection has exhausted its connection-scoped symbol dictionary. Retrying a new symbol on that sender cannot succeed. Return a pooled sender normally so the connection is retired, then borrow a fresh sender. If older published frames must be confirmed first, wait for their FSN before returning the sender. For a standalone sender, call close_drain() and check that it succeeds, then close and reopen it. Check err.in_doubt before replaying the failed input: a chunked flush may already have published an earlier prefix.

• Query timeout. QuestDBErrorCode.QueryTimeout means that a statement ran past the query_timeout_ms set in the configuration string. Do not retry a write on this error: an INSERT, UPDATE or DDL statement that timed out waiting for the table writer may still be applied afterwards. A statement that completes past its timeout returns normally rather than raising this error, so it is never reported as timed out after it took effect.

• Draining on close. Sender.close_drain waits for outstanding frames to be acknowledged before closing.

• DataFrame bulk loads. A standalone ws/wss Sender.dataframe() does not serialize into the sender’s row stream: it opens (and closes) its own poolless direct columnar connection per call, built from the sender’s own configuration (carrying its auth/TLS), and commits before returning. It has no ordering relationship with rows buffered via row() and does not flush them. Per-call connection setup is not free — batch your frames accordingly, or use questdb.connect() / QuestDB for repeated loads (its dataframe() borrows a pooled direct connection instead).

QWP/UDP

QWP/UDP (udp) uses fire-and-forget UDP datagrams for lowest-latency ingestion. It does not support authentication, TLS, or transactions. The default port is 9007. See the QWP over UDP example.

Key differences from ILP:

• No delivery guarantee. UDP datagrams may be dropped under load or network congestion. There is no retry mechanism and the server sends no acknowledgement. Use ILP/HTTP if you need reliable delivery.

• No error feedback. If a row contains invalid data (e.g. wrong column type for an existing table), the server silently drops it. With ILP/HTTP you would get an error response.

• Buffer inspection. bytes(sender) returns b'' because QWP encoding is deferred to flush. len(sender) returns an estimated size hint, not the exact serialized byte count.

• Auto-flush. auto_flush_bytes defaults to max_datagram_size (1400 by default) so that rows are flushed when the buffer approaches a single datagram’s worth of data. Rows and interval thresholds work the same as ILP.

• Datagram size limit. A single row that exceeds max_datagram_size will raise QuestDBError at flush time. Configure max_datagram_size via the constructor or configuration string.

• No protocol version. QWP has its own versioning. The protocol_version parameter and property are not applicable and will raise an error.

ILP/HTTP and ILP/TCP

ILP/HTTP is available from:

• QuestDB 7.3.10 and later.

• QuestDB Enterprise 1.2.7 and later.

ILP/HTTP also supports protocol version auto-detection.

Protocol	Protocol version auto-detection
ILP/HTTP	Yes: The client will communicate with the server using the latest version supported by both client and the server.
ILP/TCP	No: You need to configure protocol_version=N to match a version supported by the server.
QWP/UDP	N/A: QWP uses its own wire format. The protocol_version setting is not applicable.

Note

The client will disable features that require a newer protocol version than the one used to communicate with the server.

Since TCP does not block for a response it is useful for high-throughput scenarios in higher latency networks or on older versions of QuestDB which do not support ILP/HTTP quite yet.

It should be noted that you can achieve equivalent or better performance to TCP with HTTP by using multiple sender objects in parallel.

Either way, you can easily switch between the two protocols by changing:

• The <protocol> part of the configuration string.

• The port number (ILP/TCP default is 9009, ILP/HTTP default is 9000, QWP/UDP default is 9007).

• Any authentication parameters such as username, token, et cetera.

Querying data

QuestDB reads query results back over the QWP/WebSocket read endpoint. QuestDB.query returns a single-use QueryResult that streams rows as Arrow record batches:

with questdb.connect('ws::addr=localhost:9000;') as db:
    with db.query('SELECT * FROM trades WHERE ts > $1',
                  [datetime.datetime(2026, 7, 1)]) as result:
        df = result.to_pandas()

Statements with nothing to return — DDL, INSERT — go through QuestDB.execute instead: it runs the SQL, drains the result and returns the pooled connection in one call (PooledReader offers the same verb):

db.execute('CREATE TABLE IF NOT EXISTS trades ('
           '  timestamp TIMESTAMP, symbol SYMBOL, price DOUBLE'
           ') TIMESTAMP(timestamp) PARTITION BY DAY WAL')

Statement output (a COPY status row, admin-function rows, a stray SELECT) is discarded — use query() when you want it.

execute() returns the statement’s ExecDone named tuple (op_type, rows_affected), or None for a statement that streamed a result set instead. The same report is on QueryResult.exec_done once a result has been drained, and stays there after the result is closed. rows_affected is meaningful for INSERT and UPDATE, and 0 for a statement that carries no count, such as one executed at parse time; op_type is the server’s operation-type byte, carried through as an opaque int:

done = db.execute('INSERT INTO trades VALUES (now(), \'AAPL\', 1.5)')
print(done.rows_affected)

Positional bind parameters fill the $1..``$N`` placeholders — always prefer them over interpolating values into the SQL text. Supported bind types: None (SQL NULL), bool, int, float, str, datetime.datetime, TimestampMicros, TimestampNanos, datetime.date (a DATE), decimal.Decimal (a DECIMAL of the narrowest width that holds it, with its fractional digits as the scale; NaN, infinities and values beyond 76 digits or a scale of 76 raise ValueError) and uuid.UUID.

Rows of Python objects

For a row-shaped consumer — a PEP 249 cursor, dbt, plain scripting — QueryResult.iter_rows yields one tuple per row and QueryResult.columns gives (name, type_name) pairs, with neither pyarrow nor pandas involved:

with db.query('SELECT ts, symbol, price FROM trades LIMIT 3') as result:
    for name, type_name in result.columns():
        print(name, type_name)       # ts TIMESTAMP / symbol SYMBOL / ...
    for ts, symbol, price in result.iter_rows():
        print(ts, symbol, price)

type_name is the QuestDB DDL spelling (VARCHAR, TIMESTAMP_NS, IPv4, GEOHASH(8c), DECIMAL(18,2), DOUBLE[]), so it can be compared against tables() / information_schema.questdb_columns() without normalising. Two things the wire does not carry: a DECIMAL is reported at its storage width’s precision (18, 38 or 76), since only the scale is sent, and an array column is reported as DOUBLE[] / LONG[] whatever its dimensionality, where the server would spell a 2-D column DOUBLE[][]. Asking for the columns does not consume the rows, so a DB-API-style caller can read its description first, but it does rule out to_pandas() and the other columnar methods on that result. SELECT ... LIMIT 0 reads a schema without fetching anything, and the connection goes back to the pool. A result that returned rows is not at its end until they are read: iterate it, or call cancel() before closing it, to keep its connection (closing it straight after columns() drops the connection, and makes a PooledReader lease terminal). Both are empty for a statement that ships no result set — see exec_done for that outcome.

Rows stream one batch at a time and map SQL NULL to None. Timestamps are timezone-aware UTC (TIMESTAMP_NS rounded down to microseconds, since datetime has no nanosecond field), DECIMAL keeps its scale, GEOHASH comes back as the text form the server itself prints, IPv4 as ipaddress.IPv4Address, and DOUBLE[] as numpy.ndarray (LONG[] is not supported on this path). BOOLEAN, BYTE and SHORT have no NULL representation in QuestDB, so an inserted null reads back as False / 0.

This path costs an allocation per cell, so it suits small result sets — metadata lookups, test rows, schema probes. Large analytical results belong on the columnar methods below.

Per-query timeouts

Every query otherwise runs under the server-wide query.timeout (60 s by default), which is short for a long transform. query_timeout_ms in the connection string sets a default for every query on a handle, and timeout= overrides it per query (milliseconds, or a datetime.timedelta):

with questdb.connect('ws::addr=localhost:9000;query_timeout_ms=600000;') as db:
    db.execute('INSERT INTO ohlc SELECT ... FROM trades SAMPLE BY 1m')
    db.execute("CREATE TABLE trades_2025 AS (SELECT * FROM trades WHERE ts IN '2025')",
               timeout=3_600_000)

The server applies no ceiling, so a value above query.timeout is honoured. timeout=None (the default) keeps the connection string’s query_timeout_ms; timeout=0 clears it for that query. An int is milliseconds (any integer type, a numpy.int64 included); a datetime.timedelta or pandas.Timedelta works too, with a positive value below one millisecond rounded up to 1 ms.

This needs a server advertising CAP_QUERY_TIMEOUT. Against an older one a non-zero timeout is refused before anything is sent, with QuestDBErrorCode.UnsupportedServer, instead of silently running under the server default. The connection is untouched by the refusal, so raise the server’s query.timeout instead, or pass timeout=0 on the same handle or lease to clear it. (The Java client behaves differently here: it enforces the timeout itself against such a server.) On expiry the error carries QuestDBErrorCode.QueryTimeout and the pooled connection stays open and authenticated, so the next query runs on it without reconnecting.

Warning

Do not retry a write after a timeout. A statement that completes past its timeout is reported as done rather than as a timeout, precisely so a retry cannot apply it twice, and a DDL / INSERT / UPDATE that timed out waiting for the table writer may still be applied afterwards.

Columnar results

A QueryResult can be materialised with to_arrow / to_pandas or streamed batch-by-batch with iter_arrow / iter_pandas. to_arrow / iter_arrow (and to_pandas / iter_pandas with dtype_backend or types_mapper) require pyarrow; the default to_pandas / iter_pandas are pyarrow-free. It also implements the Arrow C stream PyCapsule protocol (__arrow_c_stream__), so polars and duckdb read it directly without pyarrow installed: polars.DataFrame(result) (polars 1.32 or later; older polars raises InvalidApiCall) or duckdb.from_arrow(result) (duckdb 1.2 or later). duckdb reads when you fetch, so fetch inside the with block. These reads stream the result and do not replay it: if a failover occurs after the first batch, the consumer raises its own exception type, and the message contains [FailoverWouldDuplicate]. Re-issue the query. polars before 1.37.1 raises a PanicException, which except Exception does not catch. A stream read releases the connection when the consumer reads the last batch, not when it releases the stream.

The result also implements __arrow_c_schema__, which duckdb 1.5, pa.schema(result) and polars.Schema(result) call. It reads the first batch, so after any of these calls, read the result only through the stream (pa.table(result), polars.DataFrame(result), duckdb); to_pandas() and the other readers raise InvalidApiCall, and you must run the query again to use them.

Each result is consumed once. Fully drain it, use it as a context manager (with db.query(...) as result:), or call QueryResult.close. A partially-consumed result cannot return its connection to the pool — closing it drops the connection and the pool refills on demand. To stop streaming while preserving the connection, call QueryResult.cancel, which blocks while draining to a terminal reply, then close the result (or leave its context manager).

A result, including its Arrow C stream, may be handed to a worker thread. Use a synchronized hand-off and access it from only one thread at a time; concurrent consumption, cancellation, and close are unsupported.

SYMBOL columns: to_polars / to_pandas build the categorical directly (connection dictionary interned once, no per-row remap). to_arrow / iter_arrow / __arrow_c_stream__ emit a generic Arrow form whose per-batch SYMBOL dictionary is compacted to the values each batch uses, which a generic consumer reconciles. So when the target is a polars / pandas frame, the dedicated methods avoid the re-reconciliation that polars.DataFrame(result) / to_arrow().to_pandas() pay on SYMBOL-heavy results. to_arrow().to_pandas() needs pyarrow 23 or later for SYMBOL columns, which use uint32 dictionary indices.

Reader leases

For several queries in a row, call QuestDB.reader to take a reader lease — the read-side twin of QuestDB.sender. The lease borrows one reader connection from the pool for its lifetime and runs queries on it sequentially:

with db.reader() as r:
    r1 = r.query('SELECT * FROM t1').to_pandas()
    r2 = r.query('SELECT * FROM t2',
                 reset_symbol_dict=False).to_pandas()

Each query skips the per-call pool round-trip, and because they share one connection, reset_symbol_dict=False on follow-up queries keeps the connection’s SYMBOL dictionary warm across them — skipping the re-interning of symbols the connection already knows. The default (True) keeps each result’s dictionary exactly as large as the values it uses, so materialising SYMBOL columns into pandas or polars categoricals stays compact. Queries on a lease are strictly sequential: fully drain (or close) each QueryResult before the next r.query() call. Closing a result before draining it tears down the lease’s connection, after which the lease is terminal — close it and take a fresh one. To abandon a result without losing the lease, call cancel() and then close(); cancellation drains to terminal, so the next query can reuse the connection. A lease has thread affinity: use one lease per thread, on the thread that created it (threads sharing a QuestDB handle each take their own lease). A QueryResult may be handed to a worker thread; join that worker before using the lease for its next query.

DataFrame commit and retry behavior

The same QuestDB can ingest DataFrames through the pooled columnar QWP path with QuestDB.dataframe. DataFrame ingestion always uses the direct (non-store-and-forward) column sender, independent of sf_dir.

On success, the call returns only after every row has been committed. The first successful batch on a fresh direct connection is already a commit boundary; later batches are pipelined until the final commit. A very large Arrow load adds a checkpoint about every 100 batches to keep memory bounded, and the client may checkpoint earlier if deferred capacity fills or a batch must be split to fit. The 100-batch interval therefore limits the uncommitted tail; it is not a promise that the earlier batches are safe to replay.

On a transient connection failure, the client re-sends the original, replayable DataFrame only if no batch from the call was successfully published and the failed native operation is not in_doubt. Once any batch may have committed, it raises rather than replaying from row zero and exposes in_doubt=True for the whole DataFrame call, even if the final native write alone was provably not delivered. This includes local validation and Arrow stream errors after publication, even after an internal checkpoint succeeded. This aggregation belongs to dataframe(); a low-level sender flush’s flag does not describe earlier independent flushes. The exception still means the load did not finish: an already committed prefix remains in the table, and an application-level retry of the entire DataFrame can duplicate it unless the table uses suitable DEDUP UPSERT KEYS. A consumed one-shot Arrow stream is also not replayable; when no batch could have landed that separate condition raises with in_doubt=False and asks for a fresh reader.