Bit array

Q: How deeply does Fast Python: High performance techniques for large datasets cover bit array?

Explains per-row validity bits, nullable integer storage, and pandas conversion costs, but does not detail bit-level operations.

As explained in Fast Python: High performance techniques for large datasets

A bit array is a compact, indexed data structure whose elements are individually accessible binary values.

A bit array is a compact, indexed structure whose entries hold binary values. Apache Arrow uses one validity bit per row to record whether a value is missing, keeping null metadata separate from the column’s value representation. section 7.4.2

This design lets Arrow store nullable integers as integers instead of recoding them, and makes columns with few possible states especially compact. The trade-off is conversion cost and higher memory use when the data moves into pandas. section 7.4.2

Nullable categorical columns

A CSV column such as VendorID can be converted to boolean values while its missingness remains in the separate bit array. Arrow can therefore represent the values compactly, whereas the pandas version may require a larger object representation. section 7.4.2

How Fast Python: High performance techniques for large datasets covers it

Cover of Fast Python: High performance techniques for large datasets

liveBook manning.com

Read Fast Python: High performance techniques for large datasets on liveBook Get the book at manning.com

Related concepts

missing valueused by · A missing value is a marker or condition indicating that an expected data value or reference is absent, unavailable, unknown, or inapplicable.
bloom filterpart of · A Bloom filter is a space-efficient probabilistic data structure for testing set membership that permits false positives but not false negatives.
bytescontrasts with · Bytes are an ordered sequence of values, each conventionally eight bits, used to represent binary data.

This explainer was generated by Manning's knowledge graph from the text of Fast Python: High performance techniques for large datasets. The section links point to the parts of the book it draws on.