calc_prob_coherence() takes one co-occurrence crossproduct instead of one
per topic. It used to subset the document-term matrix to a topic's top
terms, binarize that subset and cross-multiply it, once per topic. Topics
share terms heavily, so the union of every topic's top terms is far smaller
than the sum of them --- 236 distinct terms at k = 100, 918 at k = 600 on
20 Newsgroups --- and one crossproduct over that union serves every topic,
each reading its own block out of the result. 3.7x faster at k = 100 and
2.2x at k = 600; 4.0x on a 70,000-document corpus, where it falls from
1.09 s to 0.27 s. Term selection also uses a partial sort now, with ties
broken by term index so the terms chosen do not depend on how the
candidates were found.
Two large temporary allocations are gone from every fit.
summarize_topics() computed topic prevalence by building a
documents-by-topics matrix only to collapse it to a vector of topic totals;
it now takes the matrix-vector product directly, which at 70,000 documents
and k = 100 avoids a 57 MB temporary. new_tidylda() scanned theta and
beta for missing values on every fit, allocating a logical matrix the size
of each, and now checks with anyNA() first --- which stops at the first
missing value and does nothing at all when, as is almost always the case,
there are none.
Results are unchanged. Every one of these is a pure speed change:
beta, theta, lambda and the topic summary are bit-identical to 0.1.0
for the same seed, as is calc_prob_coherence() called directly.
RcppThread::LdFlags(). 0.1.1
shipped a configure script that probed for libatomic itself. RcppThread
already supplies that probe, so src/Makevars asks for the flags directly
and configure, cleanup and src/Makevars.in are gone. This also passes
-lpthread, which tidylda never had, and requires RcppThread >= 2.1.3.
Build-only: no user-visible change and no change to results.libatomic where the compiler needs it. RcppThread's
thread pool keeps each worker's loop range in an over-aligned
std::atomic. clang treats an over-aligned atomic as not lock-free and
emits a call into libatomic, which was not being linked, so the shared
object failed to load on CRAN's r-devel-linux-x86_64-debian-clang after
that machine moved to LLVM 23. A new configure script probes the
construct and adds -latomic only on toolchains that require it. This is a
build fix: no user-visible change, and no change to results.Model fitting now uses warpLDA (Chen et al., 2016, https://arxiv.org/abs/1510.08628), a Metropolis-Hastings sampler, in place of the collapsed Gibbs sampler used through 0.0.7. It alternates document-ordered and word-ordered passes over the corpus so that each pass touches only a small, cache-resident working set. On this package's benchmark corpora it is roughly 3x faster than the previous sampler single-threaded at matched quality, and roughly 23x faster end to end once threading is used. Fit quality was validated against the old sampler across a grid of corpus sizes and topic counts, under both scalar and matrix (tLDA) priors; no metric degraded.
threads is now functional in tidylda(), refit.tidylda(), and
predict.tidylda(). Reproducibility does not depend on the number of
threads: a model fitted under set.seed() is reproducible whether it was
fitted on one thread or twenty. It still defaults to 1.
tidylda() and refit.tidylda() gain mh_steps as an argument, the number of
Metropolis-Hastings proposals per token per pass.
The log_likelihood slot gained a second metric. Alongside the existing
log_likelihood column — the plug-in P(tokens | theta, beta) — there is now
a log_joint column holding the collapsed joint,
P(tokens, topics | alpha, eta), with theta and beta analytically
integrated out. This is the quantity most of the LDA literature reports. It
is always negative, it carries an implicit penalty for model complexity, and
it is the sampler's own target, which makes it the more informative of the
two for judging convergence. Neither is valid for comparing models to each
other; see ?tidylda.
This replaces an undocumented internal quantity that was computed but never surfaced — a plug-in Dirichlet density which had a sign error on both of its normalizing constants and which, being a density rather than a probability, was unbounded above and routinely positive.
There is exactly one, and it fails silently rather than raising an error, so it is worth
a moment even if you do not think you use counts in a tidylda object.
counts$Cv changed orientation, and is now sparse. It was a dense
topics-by-tokens matrix and is now a tokens-by-topics dgCMatrix. It is also
labeled, with tokens as rownames and topics as colnames, matching beta and
theta. counts$Cd is unchanged: still documents by topics, still dense.
Sparse storage shrinks it about 3x, which is roughly 19% off a whole fitted
model at k = 200 on a 4,400-token vocabulary, and more as the vocabulary
grows. counts$Cd was measured too and left dense deliberately: at 38-81%
nonzero, a sparse form would save nothing and can cost 20%.
Being a Matrix object rather than a base matrix, counts$Cv needs
Matrix::rowSums() rather than rowSums(), and as.matrix() before
anything that insists on a base matrix such as as.data.frame(). Indexing,
arithmetic and Matrix::t() all work as before.
This means model$counts$Cv[k, ] used to give topic k's counts over words
and now gives word k's counts over topics. It will not error and the
numbers will look plausible. If you index counts$Cv by topic, transpose it:
# before 0.1.0
model$counts$Cv[k, ]
# 0.1.0 and after
t(model$counts$Cv)[k, ] # or: model$counts$Cv[, k]
Models saved by earlier versions are still read correctly by posterior()
and refit() — tidylda detects the old orientation internally. That
protection does not extend to your own code reading a newly fitted model.
Both of these keep running and warn once per session. No code needs to change today.
predict(method = "gibbs") is now method = "mh". "gibbs" still works
and behaves identically; only the name was wrong, since the sampler is no
longer Gibbs. "dot" is unaffected. The default is now "mh".optimize_alpha is ignored in tidylda() and refit.tidylda(). Note
that this does change the model you get: it used to rescale alpha by
topic size every iteration, standing in for fixed-point estimation that was
never written. alpha is now fixed for the whole run. Passing TRUE warns
instead of raising an error.recover_counts_from_probs() was removed. It was never exported and had
no live call site, and its author had recorded that it returned wrong
counts. Only ::: callers are affected.Every call you could make against 0.0.7 still runs. The numbers it returns will differ.
set.seed() no longer reproduces a 0.0.7 model. The sampler is
different, so the chain is different. Fit quality was validated as
statistically equivalent across a grid of corpus sizes and topic counts
under both scalar and matrix priors, but individual models are not identical
and should not be expected to match.theta changed when alpha is a vector, as a consequence of the bug fix
below. Models fitted with a scalar alpha are unaffected.log_likelihood tibble gained a third column. Code selecting columns by
name is unaffected; code selecting by position is not.augment(), posterior() and predict() use substantially less memory.
augment() was routing lambda through tidy() and back, exploding a
k-by-vocabulary matrix into k*vocabulary rows only to collapse it again;
scaling every topic column in one block rather than one at a time; and using
apply(., 1, which.max), which copies the whole frame before iterating.
posterior() made four copies of every block of draws. Peak memory on a
128,000-token corpus at k = 100: augment(type = "prob") 684 MB to 342 MB,
augment(type = "class") 950 MB to 470 MB, and drawing from the posterior
582 MB to 374 MB. Results are unchanged.
Fitting is cheaper in two places. calc_lambda() built a k-by-k matrix and
multiplied it into beta to do what row scaling does directly — quadratic in
the number of topics for no reason — and initialization duplicated the entire
prior into a list before drawing from it. Both run on every fit.
summarize_topics() no longer sorts an entire vocabulary per topic to take
the top five terms. At a million-term vocabulary that is 4.1 seconds a topic
against 0.8.
posterior() and tidy() now refuse requests that cannot fit in memory
rather than exhausting the session. Both return one row per cell of something
that grows with topics times vocabulary, so an ordinary-looking call can ask
for billions of rows. The error reports the size, the limit, and a concrete
alternative. The ceiling is 1 GB, and
options(tidylda.max_result_size = <bytes>) raises it. Calls that previously
succeeded are unaffected; calls that previously crashed R now error.refit() no longer exhausts memory aligning vocabulary. When the base
model contained terms absent from the new data, refit() built a dense
matrix of zeros to pad the document term matrix — nrow(new_data) by the
number of missing terms, at 8 bytes a cell. On a 48,500-document corpus with
15,000 model-only terms that is 5.4 GB, allocated before sampling began, and
discarded immediately: cbind() returns a sparse matrix either way. Padding
is now sparse. Peak memory on that corpus falls from 8.6 GB to 2.1 GB.
This affected every refit() where the model contributed vocabulary, which
is the normal case for transfer learning, and it grew with the size of the
new corpus. It was present in 0.0.7.
predict() no longer materializes a prior it discards. Prediction holds
topics fixed, so the sampler never reads eta — but predict() expanded it
anyway, which for a vector prior meant two dense k-by-vocabulary matrices per
call. Also present in 0.0.7.
Fixed the calculation of theta when alpha is asymmetric. The prior was
added along the wrong axis of the document-topic count matrix, so instead of
alpha[k] being added to topic k, the values were recycled diagonally
across topics. Models fitted with a scalar alpha are unaffected, since
every entry is then equal; models fitted with a vector alpha — including
output from refit.tidylda() — will now report different, correct values of
theta.
Fixed a miscalculation in the log likelihood reported by tidylda() and
refit.tidylda(). A normalizing denominator was accumulated across topics
instead of being reset for each topic, so the topic-word probabilities used
in the calculation were incorrectly scaled. Note that this changes the log
likelihood values reported for a given model relative to previous versions.
Model fitting itself is unaffected.
Removed a hardcoded number of topics in an internal call from tidylda() to
initialize_topic_counts(). The value was unused, so model results are
unaffected.
Fixed an internal call in predict.tidylda() that passed an argument
positionally into the wrong parameter of new_tidylda(). The argument was
unused on the prediction path, so predictions are unaffected.
Tests using the 'quanteda' package are now skipped when it is not installed, consistent with its status as a suggested package.
testthat::skip_if_not_installed('tm') in offending test.predict.tidylda outputsconvert_dtm to not use functions deprecated as of
Matrix 1.4-2tidylda where data not returned even if user specifies return_data = TRUEtidylda:::recover_counts_from_probstidylda with large data sets.refit.tidylda when fine tuning on only
one document.refit.tidylda when beta from a previous model is used
as the prior. Miscalculation only affected multiple sequential calls to refittextmineR package.This is the first released version of tidylda!