Randomization does not help searching predecessors

At STOC'06, we presented a new technique for proving cell-probe lower bounds for static data structures with deterministic queries. This was the first technique which could prove a bound higher than communication complexity, and it gave the first separation between data structures with linear and polynomial space. The new technique was, however, heavily tuned for the deterministic worst-case, demonstrating long query times only for an exponentially small fraction of the input. In this paper, we extend the technique to give lower bounds for randomized query algorithms with constant error probability.

Our main application is the problem of searching predecessors in a static set of n integers, each contained in a l-bit word. Our trade-off lower bounds are tight for any combination of parameters. For small space, i.e. n^1+o(1), proving such lower bounds was inherently impossible through known techniques. An interesting new consequence is that for near linear space, the classic van Emde Boas search time of O(lg l) cannot be improved, even if we allow randomization. This is a separation from polynomial space, since Beame and Fich [STOC'02] give a predecessor search time of O(lg l/lg lg l) using quadratic space.

We also show a tight Ω(lg lg n) lower bound for 2-dimensional range queries, via a new reduction. This holds even in rank space, where no superconstant lower bound was known, neither randomized nor worst-case. We also slightly improve the best lower bound for the approximate nearest neighbor problem, when small space is available.