batch.hpp File Reference

Batch generation: item seeds, parallel and streaming runs, and uniqueness, shared by every generator library. More...

#include "rng.hpp"
#include <algorithm>
#include <concepts>
#include <cstddef>
#include <cstdint>
#include <exception>
#include <functional>
#include <limits>
#include <optional>
#include <stdexcept>
#include <string>
#include <system_error>
#include <thread>
#include <type_traits>
#include <typeindex>
#include <typeinfo>
#include <unordered_set>
#include <utility>
#include <vector>

Go to the source code of this file.

Classes
class	dasmig::core::unique_rule
	Makes a batch unique by a key (an email, a username, ...). More...
struct	dasmig::core::batch_options
	How a batch is generated. Requests keep describing one item. More...

Typedefs
template<typename Make >
using	dasmig::core::detail::batch_item_t = std::remove_cvref_t< std::invoke_result_t< const Make &, std::uint64_t > >

Functions
constexpr std::uint64_t	dasmig::core::item_seed (std::uint64_t batch_seed, std::uint64_t index) noexcept
	The seed of item index in a batch.
constexpr std::uint64_t	dasmig::core::attempt_seed (std::uint64_t batch_seed, std::uint64_t index, std::uint64_t attempt) noexcept
	The seed of attempt attempt of item index, for a batch that must be unique.
std::size_t	dasmig::core::detail::resolve_threads (std::size_t requested, std::size_t work) noexcept
template<typename Make , typename Item >
void	dasmig::core::detail::fill_window (const Make &make, std::uint64_t batch_seed, std::uint64_t first, std::size_t n, std::size_t threads, std::vector< std::optional< Item > > &slots)
template<typename Make , typename Sink >
void	dasmig::core::detail::run_batch_windows (const batch_options &options, std::uint64_t batch_seed, const Make &make, Sink &sink, std::uint64_t window_items)
template<typename Make > requires std::invocable<const Make&, std::uint64_t>
std::vector< detail::batch_item_t< Make > >	dasmig::core::run_batch (const batch_options &options, std::uint64_t batch_seed, const Make &make)
	Generate a batch and return it.
template<typename Make , typename Sink > requires std::invocable<const Make&, std::uint64_t>
void	dasmig::core::for_each_batch (const batch_options &options, std::uint64_t batch_seed, const Make &make, Sink &&sink)
	Generate a batch and pass each item to sink, in index order, without holding the whole batch.

Detailed Description

Batch generation: item seeds, parallel and streaming runs, and uniqueness, shared by every generator library.

Author

Diego Dasso Migotto (diegomigotto at hotmail dot com)

A batch is count results generated from one batch seed. Item i is exactly what the single-call getter returns for item_seed(batch_seed, i), so any item replays on its own, any range is computed directly, and the result does not depend on how the work is divided between threads.

Definition in file batch.hpp.

Typedef Documentation

batch_item_t

template<typename Make >

using dasmig::core::detail::batch_item_t = typedef std::remove_cvref_t<std::invoke_result_t<const Make&, std::uint64_t> >

Definition at line 135 of file batch.hpp.

Function Documentation

attempt_seed()

constexpr std::uint64_t dasmig::core::attempt_seed ( std::uint64_t  batch_seed, std::uint64_t  index, std::uint64_t  attempt  )

constexprnoexcept

The seed of attempt attempt of item index, for a batch that must be unique.

Attempt 0 is the plain item_seed(), so items that do not collide equal the single call. Attempt k > 0 is item_seed(item_seed(batch_seed, index), k).

Parameters

batch_seed	Seed of the whole batch.
index	Index of the item.
attempt	Attempt number, starting at 0.

Returns

The seed to generate that attempt with.

Definition at line 64 of file batch.hpp.

fill_window()

template<typename Make , typename Item >

void dasmig::core::detail::fill_window	(	const Make &	make,
		std::uint64_t	batch_seed,
		std::uint64_t	first,
		std::size_t	n,
		std::size_t	threads,
		std::vector< std::optional< Item > > &	slots
	)

Definition at line 157 of file batch.hpp.

for_each_batch()

template<typename Make , typename Sink >
requires std::invocable<const Make&, std::uint64_t>

void dasmig::core::for_each_batch	(	const batch_options &	options,
		std::uint64_t	batch_seed,
		const Make &	make,
		Sink &&	sink
	)

Generate a batch and pass each item to sink, in index order, without holding the whole batch.

At most a few thousand items per thread are alive at once (plus the keys of a unique batch). sink runs on the calling thread and receives the item's index (including options.first) and the item.

Template Parameters

Make	Callable taking a std::uint64_t seed and returning an item.
Sink	Callable taking (std::uint64_t index, Item&).

Parameters

options	Count, first index, threads and uniqueness.
batch_seed	Seed of the whole batch.
make	Generates one item from its seed.
sink	Receives the items.

Exceptions

The	same errors as run_batch(), and whatever sink throws.

Definition at line 321 of file batch.hpp.

item_seed()

constexpr std::uint64_t dasmig::core::item_seed ( std::uint64_t  batch_seed, std::uint64_t  index  )

constexprnoexcept

The seed of item index in a batch.

rng{batch_seed}.split(index).seed(), computed without building an engine.

Parameters

batch_seed	Seed of the whole batch.
index	Index of the item (including batch_options::first).

Returns

The seed to give the single-call getter for that item.

Definition at line 48 of file batch.hpp.

resolve_threads()

std::size_t dasmig::core::detail::resolve_threads ( std::size_t  requested, std::size_t  work  )

inlinenoexcept

Definition at line 138 of file batch.hpp.

run_batch()

template<typename Make >
requires std::invocable<const Make&, std::uint64_t>

std::vector< detail::batch_item_t< Make > > dasmig::core::run_batch	(	const batch_options &	options,
		std::uint64_t	batch_seed,
		const Make &	make
	)

Generate a batch and return it.

Item i is make(item_seed(batch_seed, options.first + i)). make runs on several threads when options.threads allows, so it must be callable concurrently (the libraries' const seeded getters are).

auto names = dasmig::core::run_batch({.count = 1000, .threads = 4}, 7,
    [&](std::uint64_t seed) { return gen.get_name(request, seed); });

Template Parameters

Make	Callable taking a std::uint64_t seed and returning an item.

Parameters

options	Count, first index, threads and uniqueness.
batch_seed	Seed of the whole batch.
make	Generates one item from its seed.

Returns

The items, in index order.

Exceptions

std::invalid_argument	If the options are inconsistent.
std::runtime_error	If a unique batch runs out of attempts.
Whatever	make throws, for the lowest failing index.

Definition at line 292 of file batch.hpp.

run_batch_windows()

template<typename Make , typename Sink >

void dasmig::core::detail::run_batch_windows	(	const batch_options &	options,
		std::uint64_t	batch_seed,
		const Make &	make,
		Sink &	sink,
		std::uint64_t	window_items
	)

Definition at line 209 of file batch.hpp.